From unknown Mon Jun 16 23:27:04 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#55907 <55907@debbugs.gnu.org> To: bug#55907 <55907@debbugs.gnu.org> Subject: Status: VFIO kernel module fails to capture PCI device Reply-To: bug#55907 <55907@debbugs.gnu.org> Date: Tue, 17 Jun 2025 06:27:04 +0000 retitle 55907 VFIO kernel module fails to capture PCI device reassign 55907 guix submitter 55907 "Nick Zalutskiy" severity 55907 normal thanks From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 11 09:40:49 2022 Received: (at submit) by debbugs.gnu.org; 11 Jun 2022 13:40:49 +0000 Received: from localhost ([127.0.0.1]:52206 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o01M8-00026T-C9 for submit@debbugs.gnu.org; Sat, 11 Jun 2022 09:40:48 -0400 Received: from lists.gnu.org ([209.51.188.17]:40738) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o01M5-00026K-Tv for submit@debbugs.gnu.org; Sat, 11 Jun 2022 09:40:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:44846) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o01M4-00032v-Ox for bug-guix@gnu.org; Sat, 11 Jun 2022 09:40:45 -0400 Received: from out2-smtp.messagingengine.com ([66.111.4.26]:45857) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o01M1-0001l0-LZ for bug-guix@gnu.org; Sat, 11 Jun 2022 09:40:43 -0400 Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 75BD65C00EA for ; Sat, 11 Jun 2022 09:40:39 -0400 (EDT) Received: from imap43 ([10.202.2.93]) by compute2.internal (MEProxy); Sat, 11 Jun 2022 09:40:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=const.fun; h=cc :content-type:date:date:from:from:in-reply-to:message-id :mime-version:reply-to:sender:subject:subject:to:to; s=fm1; t= 1654954839; x=1655041239; bh=OFPKtNqzSmK68K4uUHtg6L8C6NRQ2QV7m05 HA6cm9hY=; b=bk3wh5x0Tj3niWJPpN5j5x1nX46gb6JGdEGwYvR/Qm+xRHG+zYu OfF+4qY/hZ8rvisV0TfSiwlsFy6px+Sz1t5g64s1Ax7kHnSnTHpwCe6dNqCHbjZQ XTmmUIwvAfIihO53uoxxq3FG84379Rklmq48zTWR5tymK7qk2YARL2BwK5+FW+Oj GSFsi0MpmuoGwr6EoZsPhWh8T0eYgU9rX3cXL9IuAJdPlBYWcnQv/5FmpjcWnIvs NDlujdsC94ntkytr5NliFADKkUannQz5rI1LcjvD4r2SgP2fOV9qmUOlhZBWECKg LOK30L0ttZX7qJ6vlg8h6ZsN6ZSMmMY9mIQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:message-id:mime-version :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1654954839; x= 1655041239; bh=OFPKtNqzSmK68K4uUHtg6L8C6NRQ2QV7m05HA6cm9hY=; b=X tt6Kf8boA33/VuJfKwZQDuPqNT3jqSpBUe9f8sMsG3eCCiXPrleWF8XyVr1iFQi9 z2wqDVJY+AUdkfVjxZnHsDYBUm1NT8Ov9ZCFtSn7VebESQhsO4zjGRTSd2iHgjDE ccQnvzSfKU141U0uSL4GvrhNaS5FNyWyy8IpR7o10X2lEVf6jOFedm0+mctEurBl 4AFwrtT2nfurHIN7lQf/FJ5TnUakKgLM7j9qLtcj0O10URebLBO845i5Hu3hB0eb bBmvXu4Tu/ynE0mOH/76fQjpjMYAetz1YDmEOyOMSA8MEWAvzlkEq9EoKYcoLy8R pL5+fvD0ykjCEXUhcFOZw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedruddufedgieehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfffhffvufgtsegrtderre erredtnecuhfhrohhmpedfpfhitghkucgkrghluhhtshhkihihfdcuoehnihgtkhestgho nhhsthdrfhhunheqnecuggftrfgrthhtvghrnhepffethfdthefgkedugeetieekueevfe fhgfeiffetgfdvgeehleeitefhheehfeeinecuffhomhgrihhnpehgihhthhhusgdrtgho mhdprghrtghhlhhinhhugidrohhrghdpkhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrh fuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepnhhitghksegtohhnshhtrdhf uhhn X-ME-Proxy: Feedback-ID: i25a94707:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 276A62D40070; Sat, 11 Jun 2022 09:40:39 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.7.0-alpha0-692-gb287c361f5-fm-20220603.003-gb287c361 Mime-Version: 1.0 Message-Id: Date: Sat, 11 Jun 2022 09:40:18 -0400 From: "Nick Zalutskiy" To: bug-guix@gnu.org Subject: VFIO kernel module fails to capture PCI device Content-Type: multipart/alternative; boundary=88555654cab94e5396cdcfa3e24335ec Received-SPF: pass client-ip=66.111.4.26; envelope-from=nick@const.fun; helo=out2-smtp.messagingengine.com X-Spam_score_int: 17 X-Spam_score: 1.7 X-Spam_bar: + X-Spam_report: (1.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FROM_SUSPICIOUS_NTLD=0.499, FROM_SUSPICIOUS_NTLD_FP=1.997, HTML_MESSAGE=0.001, PDS_OTHER_BAD_TLD=1.997, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-Spam-Score: 2.5 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hello all, I am trying to capture my graphics card at initrd, using vfio, to later pass it through to a virtual machine. Judging by dmesg, the VFIO module does load early, however, the card is not captured at th [...] Content analysis details: (2.5 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [209.51.188.17 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 0.7 SPF_NEUTRAL SPF: sender does not match SPF record (neutral) 1.6 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: const.fun (fun)] 0.0 HTML_MESSAGE BODY: HTML included in message 2.0 FROM_SUSPICIOUS_NTLD_FP From abused NTLD 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD -0.0 T_SCC_BODY_TEXT_LINE No description available. X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) --88555654cab94e5396cdcfa3e24335ec Content-Type: text/plain Hello all, I am trying to capture my graphics card at initrd, using vfio, to later pass it through to a virtual machine. Judging by dmesg, the VFIO module does load early, however, the card is not captured at that point and the amdgpu driver is later loaded instead. This is what I have in my `operating-system` config: > (kernel-arguments '("iommu=pt" "vfio-pci.ids=1002:73bf")) > (initrd-modules (cons* "vfio_pci" "vfio" "vfio_iommu_type1" "vfio_virqfd" %base-initrd-modules)) There are two video cards in the system, both AMD, but different models. The video card of interest is in a separate IOMMU group and the : combination is correct for my machine. Best I can tell, vfio-pci.ids argument is not propagated to the module by initramfs. See the following: Searching online I came up against a GitHub issue for a different initramfs generator that exhibited the same symptoms: VFIO module was loaded, kernel arguments were correct, yet the card was not captured by the vfio driver. The maintainer there did a great job tracking down and fixing the issue and came up with this insight https://github.com/anatol/booster/issues/20#issuecomment-808956316 > After reading kmod code I found that kernel does not use cmdline params for loadable modules. It was surprising for me. Instead it is expected that userspace handles cmdline parsing and provides required module params explicitly. Another way to attach the correct driver to the gpu is to run a script at initrd, which I don't know how to accomplish with Guix. This approach has the advantage of working with two identical video cards (or disks, etc) See https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Using_identical_guest_and_host_GPUs I tried following the kernel docs to rebind a different driver after boot, but I believe this doesn't work for video cards, and hasn't worked for me. Any help is greatly appreciated! Links: Kernel docs for vfio https://www.kernel.org/doc/html/latest/driver-api/vfio.html Arch guide for GPU passthrough https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF Thank you! -Nick --88555654cab94e5396cdcfa3e24335ec Content-Type: text/html Content-Transfer-Encoding: quoted-printable
Hello all,
<= /div>

I am trying to capture my graphics card at init= rd, using vfio, to later pass it through to a virtual machine. Judging b= y dmesg, the VFIO module does load early, however, the card is not captu= red at that point and the amdgpu driver is later loaded instead.

This is what I have in my `operating-system` confi= g:

(kernel-argume= nts '("iommu=3Dpt" "vfio-pci.ids=3D1002:73bf"))
  (in= itrd-modules (cons* "vfio_pci" "vfio" "vfio_iommu_type1" "vfio_virqfd" %= base-initrd-modules))

There ar= e two video cards in the system, both AMD, but different models. The vid= eo card of interest is in a separate IOMMU group and the <vendor id&g= t;:<device id> combination is correct for my machine.

Best I can tell, vfio-pci.ids argument is not propagate= d to the module by initramfs. See the following:

Searching online I came up against a GitHub issue for a different = initramfs generator that exhibited the same symptoms: VFIO module was lo= aded, kernel arguments were correct, yet the card was not captured by th= e vfio driver. The maintainer there did a great job tracking down and fi= xing the issue and came up with this insight https://github.com/a= natol/booster/issues/20#issuecomment-808956316

After reading kmod code I found that k= ernel does not use cmdline params=20 for loadable modules. It was surprising for me. Instead it is expected=20 that userspace handles cmdline parsing and provides required module=20 params explicitly.

Another way= to attach the correct driver to the gpu is to run a script at initrd, w= hich I don't know how to accomplish with Guix. This approach has the adv= antage of working with two identical video cards (or disks, etc) See https://wiki.archlinux.org/title/PCI_pass= through_via_OVMF#Using_identical_guest_and_host_GPUs
<= br>
I tried following the kernel docs to rebind a different dr= iver after boot, but I believe this doesn't work for video cards, and ha= sn't worked for me.

Any help is greatly app= reciated!

Links:
Kernel docs = for vfio

Arch guide for GPU passthrou= gh

Thank you!

-Nick

--88555654cab94e5396cdcfa3e24335ec-- From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 08 08:43:26 2023 Received: (at 55907) by debbugs.gnu.org; 8 Sep 2023 12:43:26 +0000 Received: from localhost ([127.0.0.1]:42524 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qeapZ-0003mP-Ng for submit@debbugs.gnu.org; Fri, 08 Sep 2023 08:43:26 -0400 Received: from mail-lf1-x131.google.com ([2a00:1450:4864:20::131]:46348) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qeX67-0000W1-Lh for 55907@debbugs.gnu.org; Fri, 08 Sep 2023 04:44:16 -0400 Received: by mail-lf1-x131.google.com with SMTP id 2adb3069b0e04-500c7796d8eso3048504e87.1 for <55907@debbugs.gnu.org>; Fri, 08 Sep 2023 01:44:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694162647; x=1694767447; darn=debbugs.gnu.org; h=content-disposition:mime-version:message-id:subject:to:from:date :from:to:cc:subject:date:message-id:reply-to; bh=fDXW2vaB+qPTOZFwBeFq09M9BfulFrw4UnvLtN4aGgw=; b=eVRjDQpfAd3fyCPvtV6mYt/3LETj3PM3viPA3BzKB3oy/LcsxbtcHi2+3r8xPwR/ZS XQfZU7MakJCGWJ7EXn49AAlf+9LAGOoL6Vny6R+Ut8MhIZtdEBZjY/ZB66+CNv5gCz4w OsBMLHlPKgYehfoyR/OgVga2BLNupUZkGNoOO1p/3pwlB89xJplqYaDIZVSd16Uo0xtP FgAB2eR13uEhUKEP7CiYKN1B6R8y+hEk85etI03cWlc16NAbHMueGOAqdxo92c1XFji2 zA9CttPVXqyObrr9R/FoOO0qVFgNaEGpzSoC2eSg+xQqC9gJbwgHOgEWLfVVueUdSB8F RSuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694162647; x=1694767447; h=content-disposition:mime-version:message-id:subject:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fDXW2vaB+qPTOZFwBeFq09M9BfulFrw4UnvLtN4aGgw=; b=ksctOthBKa2SH8l/v2JUpRy0evVWaD/khmGhYmtMukftOONzvGp2+voNugJ6zZDqFz Qp1G2UHv0llGXKhwKyVP56dILYr9ifRIkZvZzX2YUtcPlyUZX6Vp2vddhZz6KP0ODGvR ExsUIn8d7LIXXEaA32rNpkVOiKEa9ngE9uQh+D5bp1tGk+bHE624G1/4fCdXfSBXnZQl kFZ+OUeLO7E0anZdCh47oiOVvJcpe351OJWTpGTeWw5K+NTrbOYNZyDZ8+Ypj0/3qjEx JMKjuGiA1PZDBOUSbpd6ULr/wa6VdErGWrDUKQG8k7QoLEpodGRtgYwDsOGcU7sBMt0I lgxg== X-Gm-Message-State: AOJu0Yx57wVPiF/iyybhXVJ+KdcIUOISQudj9wsB7HzuAHKZfq6+0I2V 3FHIg73MxNmKQznDnPInqtg/FQYtLiU= X-Google-Smtp-Source: AGHT+IEi1GAMSw+oygPfFeSqdKEbMhoyiBMRKsi1DZkhlV1nTbrr2krJm9jB7WMe8xghfrRbakInkA== X-Received: by 2002:ac2:4f0a:0:b0:4fa:21d4:b3ca with SMTP id k10-20020ac24f0a000000b004fa21d4b3camr1448821lfr.2.1694162647289; Fri, 08 Sep 2023 01:44:07 -0700 (PDT) Received: from localhost (78.79-160-60.customer.lyse.net. [79.160.60.78]) by smtp.gmail.com with ESMTPSA id x11-20020ac25dcb000000b004fe37199b87sm214561lfq.156.2023.09.08.01.44.05 for <55907@debbugs.gnu.org> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Sep 2023 01:44:06 -0700 (PDT) Date: Fri, 8 Sep 2023 10:43:57 +0200 From: Lars Rustand To: 55907@debbugs.gnu.org Subject: VFIO kernel module fails to capture PCI device Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 55907 X-Mailman-Approved-At: Fri, 08 Sep 2023 08:43:23 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello Nick, Did you ever figure this out? I am struggling with the same problem. Thank you, - Lars From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 05 00:28:56 2024 Received: (at 55907) by debbugs.gnu.org; 5 Jul 2024 04:28:56 +0000 Received: from localhost ([127.0.0.1]:43349 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sPaZ5-0003ZY-E3 for submit@debbugs.gnu.org; Fri, 05 Jul 2024 00:28:56 -0400 Received: from mail-4322.protonmail.ch ([185.70.43.22]:30443) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sPU94-00011m-7r for 55907@debbugs.gnu.org; Thu, 04 Jul 2024 17:37:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protonmail.com; s=protonmail3; t=1720129049; x=1720388249; bh=1Vvcneqpzkq6DMTG6vwNLTjVJvOPVUdAB04JipMpacU=; h=Date:To:From:Subject:Message-ID:Feedback-ID:From:To:Cc:Date: Subject:Reply-To:Feedback-ID:Message-ID:BIMI-Selector; b=vDTrqQXWAhmClj8/Xwym1Zzp2SGiCCOd9XxeT9zaveR5VX4iGKNZ2WtSSvKnVy0ip U0RbFHz18JGd3O7w9r+iWY5JB/YU16dWr7l+015qrTRcf6sRrPKoqPnkc2ziRbakBi xHc6zQWMb4aCFRcJ91rKw2+1d3cAFK9OYwKQlSPgDvpnm5pKSZTyGRKZJ6p3SCnh+J 2Jaf7r6FQ7Pi+tZScUm2GR17YlJAcIlG1wypob8bAapzd8xvzMKQyuaJybb9wSjjqP PWO7gANkfO2wYa7LarS7PQsogNFYp9xU27PLUYt7QWPOtRSkuQo+FYOqekwVOBByA7 qtcJdFLXedLVw== Date: Thu, 04 Jul 2024 21:37:24 +0000 To: "55907@debbugs.gnu.org" <55907@debbugs.gnu.org> From: =?utf-8?Q?Nikola_Brkovi=C4=87?= Subject: VFIO kernel module fails to capture PCI device Message-ID: Feedback-ID: 32797663:user:proton X-Pm-Message-ID: e4f19ad6230013e445430ba64e9a367962531ef6 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 55907 X-Mailman-Approved-At: Fri, 05 Jul 2024 00:28:54 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) I have managed to get VFIO working by creating a service of boot-service-ty= pe which overrides the GPU driver with vfio-pci and binds the GPU to VFIO: >(simple-service 'vfio-override=C2=A0boot-service-type >=C2=A0 =C2=A0 '(and (call-with-output-file "/sys/bus/pci/devices/0000:04:0= 0.0/driver_override" >=C2=A0 =C2=A0 (lambda (p) >=C2=A0 =C2=A0 =C2=A0 (display "vfio-pci" p))) >=C2=A0 (call-with-output-file "/sys/bus/pci/drivers/vfio-pci/new_id" >=C2=A0 =C2=A0 (lambda (p) >=C2=A0 =C2=A0 =C2=A0 (display "1002 665f" p))) >=C2=A0 ) >) Sorry for the hard-coded IDs, you should replace them with your own. You mi= ght need to unbind the GPU's audio card from its driver as well, after you'= re fully booted. QEMU will refuse to pass-through the GPU if the audio card= is in the same IOMMU group and not using vfio-pci. In my case, the service runs early enough in the boot process where amdgpu = has not initialized the GPU yet. There might be a better way to accomplish = this, I'm still new to Guix and Scheme. Thanks, Nikola