| Mrfai (~lange@kueppers.informatik.uni-koeln.de) joined #fai. |
| verwilst (~verwilst@router.begen1.office.netnoc.eu) joined #fai. |
| siggg (~jean@tra78-3-82-244-85-196.fbx.proxad.net) joined #fai. |
| siggg (~jean@tra78-3-82-244-85-196.fbx.proxad.net) left irc: |
| allee-k (~allee-k@alleehol.mpe.mpg.de) joined #fai. |
| raphink (~quassel@193.252.149.222) joined #fai. |
| Mrfai (~lange@kueppers.informatik.uni-koeln.de) left irc: Quit: leaving |
| Mrfai (~lange@134.95.9.136) joined #fai. |
| 10:10 allee | glance: I'm working on lucid pkgs. I've only tested demo install on Dell Optiplex 780/760. Right now, on an IBM blade I get a kernel panic :( That why I'm here now ;) |
| 10:51 mgoetze | mikap: is there no aufs-modules-2.6.33-grml64? integrated in main kernel package now? |
| 10:51 mikap | mgoetze: integrated in 2.6.33-grml, yes |
| 10:51 mikap | and 2.6.33-grml64 of course as well |
| 10:52 mgoetze | mikap: ok, well, i keep having this weird problem where fai aborts with 2.6.33 because the mountpoint /var/lib/fai/config supposedly doesn't exist |
| allee-k (~allee-k@alleehol.mpe.mpg.de) left irc: Remote host closed the connection |
| 10:52 mgoetze | mikap: /var/lib is totally empty except for /var/lib/discover, whereas on the nfs server .../filesystem.dir/var/lib is pretty full |
| 10:53 mgoetze | (meanwhile, usb keyboard isn't working on poweredge r210 with 2.6.28-grml :( ) |
| 10:53 mikap | mgoetze: Mrfai and me are working in those minutes on this issue |
| 10:53 mgoetze | ah ok |
| 10:53 mikap | not a 2.6.33 issue but a live-initramfs with regards to the noxino mountoption |
| 10:54 mgoetze | well, please let me know what i have to do to fix it once you know :) |
| 10:55 mikap | ok |
| 10:55 mikap | for now you can fall back to older live-initramfs version, JFYI |
| allee-k (~allee-k@alleehol.mpe.mpg.de) joined #fai. |
| 10:59 mgoetze | mikap: what would work? 1.157.4-1grml or 1.173.1? |
| 11:00 mikap | mgoetze: have you 5 minutes for a test? then i'd welcome giving http://grml.org/debug/live-initramfs_1.177.1-1grml.02_all.deb a shot, the safe option should be live-initramfs 1.173.1-1 |
| 11:02 mikap | and you've to rebuild initrd manually sadly because the according code seems to work only since a git commit in initramfs-tools on 25rd of march |
| 11:03 mgoetze | mikap: hm are you sure? it says |
| 11:03 mgoetze | Processing triggers for initramfs-tools ... |
| 11:03 mgoetze | update-initramfs: Generating /boot/initrd.img-2.6.33-grml64 |
| 11:04 mikap | mgoetze: ok, even better - i know i've merged code that should address it, not sure what problem Mrfai had |
| 11:05 mgoetze | mikap: ok, i'm off to try booting it, bbiab :) |
| 11:06 mikap | mgoetze: ok great :) |
| 11:11 mgoetze | mikap: same error :( |
| 11:12 mikap | mgoetze: fsck |
| 11:20 allee | #$%@ I get a kernel panic in forgot_original_parent (I assume when trying remount the NFSroot) IBM blade (xeon) with 3.3.4 with a faiserver on ubuntu lucid |
| 11:20 allee | DELL Optiplex 760, 780 (core 2) don't have the problem. Any hint for a magic kermel param? |
| siggg (~jean@tra78-3-82-244-85-196.fbx.proxad.net) joined #fai. |
| 12:06 Mrfai | allee: DONT_BUY_DELL=1 |
| 12:06 mgoetze | Mrfai: pfff... my dell hardware mostly works just fine with grml kernels :) |
| 12:06 mgoetze | DONT_USE_UBUNTU=kernel |
| 12:07 Mrfai | mika: no success with live-initramfs_1.177.1-1grml.02_all.deb |
| 12:17 mikap | Mrfai: fsck |
| 12:18 mgoetze | Mrfai: can i use "ifclass AMD64" in nfsroot-hooks? |
| 12:21 Mrfai | mgoetze: not sure, I never used hooks in make-fai-nfsroot. Just try it. |
| 12:21 mgoetze | ok |
| 12:22 Mrfai | mgoetze: Can you email me your output of make-fai-nfsroot -v. Mine does not build the initrd automatically when using the 2.6.33-grml kernel |
| 12:23 mgoetze | Mrfai: i had the same problem last time so i am still using the hook. i'm about to make a new run with 1.173 and will email you the output |
| 12:23 Mrfai | mgoetze: ifclass is not available in m-f-n since it doues not source /usr/lib/fai/subroutines (which defines ifclass). |
| 12:24 mgoetze | oh too bad :( any other way i could check? custom variable in make-fai-nfsroot.conf ? |
| 12:26 Mrfai | you can source subroutines in your hook |
| 12:26 mgoetze | actually i'm thinking maybe i will write something into make-fai-nfsroot.conf anyway since the duplication of config changes is starting to annoy me ;) |
| 12:27 mikap | Mrfai: have you diffed the log of my current l-i and the working one again? |
| 12:30 mgoetze | Mrfai: i can use $cfdir in make-fai-nfsroot.conf, right? :) |
| 12:32 allee | Mrfai: You mean DONT_BUY_IBM=1 ;) Dell Optiplex are working fine with lucid fai :) IBM blades make trouble. I'm now in TRY_A_SUN=1 ;) |
| 12:32 allee | as google didn't help :( |
| 12:33 mikap | allee: what troubles do you have with IBM blades? |
| 12:34 Mrfai | mikap: I will do the diff now |
| 12:34 mikap | Mrfai: thanks, because i really don't see the relevant difference in the code :-/ |
| 12:34 allee | mikap: [13:20] <allee> #$%@ I get a kernel panic in forgot_original_parent (I assume when trying remount the NFSroot) IBM blade (xeon) with 3.3.4 with a faiserver on ubuntu lucid |
| 12:35 mikap | allee: hm |
| 12:36 allee | It's after livefs premount done, lots of kern msg. then sda is found, unknown partition table (as it's a black disk) then the kernel panic. I've tried with linux-image and linux-server from lucid. same problem |
| 12:38 mikap | allee: black == blank? |
| 12:38 allee | mikap: ups, right blank, never used, fresh, untouched |
| 12:39 mikap | allee: ok, do you have initcall_debug nomce in your kernelcmdline? |
| 12:39 mikap | oh and "debug" |
| 12:39 mikap | or debug=1 |
| 12:41 allee | mikap: I'll try debug=1 initcall_debug=1 ... |
| 12:48 allee | mikap: kernel panic screen: http://imagebin.ca/view/diNMPr.html |
| 12:48 mgoetze | Mrfai: post fuer dich |
| Action: allee should better find out how to get serial output of the blade |
| 12:49 mikap | allee: kernel cmdline looks like? |
| 12:49 allee | append initrd=initrd.img-2.6.32-18-server ip=dhcp root=/dev/nfs nfsroot=/srv/fai/nfsroot-lucid64 boot=live debug=1 initcall_debug=1 FAI_FLAGS=verbose,sshd,createvt FAI_ACTION=install |
| Action: mgoetze goes to see whether 2.6.33/1.173.1 will boot :) |
| 12:56 mgoetze | mikap: with 2.6.33/1.173.1 i get Mounting root filesystem... kernel panic: attempted to kill init |
| 12:56 mikap | mgoetze: huh? why does it work for Mrfai? |
| 12:56 mgoetze | dunno, maybe hardware |
| Action: mgoetze will try 1.157.4-1grml.01 |
| 12:58 mikap | allee: strange, never saw that one on ibm blades, trying "mpt_debug_level=8 mptbase.mpt_debug_level=8" as additional kernel cmdline could be worth a try |
| 13:00 mikap | allee: you've the most recent LSI firmware applied to the disks and the blade itself is up2date as well? |
| 13:01 allee | mikap: hardware is ~ 1 year old. No firmware updates yet. mpt is 6.20... afair form boot msgs |
| 13:03 mikap | allee: do other kernel versions work? |
| 13:04 allee | mikap: no helpful debug code with mpt*debug on kern line: http://imagebin.ca/view/gaZ6vjW.html |
| 13:06 mikap | allee: that's bad, i'd give another kernel version a shot |
| 13:06 allee | mikap: not tried. I started using fai lucid yesterday. I'll give SUN X2100. X4100 and Dell R410 a try. Let's see if they work ... |
| 13:06 mikap | allee: ah ok |
| 13:06 mikap | i've just experience with debian's and grml's kernel and ibm blades and they usually work just fine, jfyi |
| Action: mgoetze goes to test 2.6.33/1.157.4 |
| 13:11 allee | mikap: I know that a SUN X4100 and Dell Latitude E8500 laptop run fine with this kernel (installed via CD, upgraded and rebooted). Optiplex 760/780 fai install with this kernel without problem |
| 13:11 mikap | allee: ah |
| 13:11 mikap | so either an LSI controller issue or really some strange vodoo ;) |
| 13:13 allee | mikap: but this aufs business seem to be fragile. Not the first time here that a (k)ubuntu CDROM works fine, but FAI nfsboot fails with same kernel |
| 13:14 mikap | allee: well yeah the aufs1->aufs2 switch is a bit PITA |
| 13:15 mgoetze | once again, kernel panic :( |
| 13:17 mgoetze | mikap: any more hopes for a new 1.177 version? |
| 13:19 mikap | mgoetze: i don't even know the root of the problem as the one part of the code that could be the reason doesn't seem to be it, I've an idea but it was destroyed by Mrfai's tests because he can successfully use 2.6.33 whereas my idea would mean that it won't work with 2.6.33 at all |
| 13:19 mikap | so i need the feedback of Mrfai regarding his tests and the resulting logs (and their diffs) |
| allee-k (~allee-k@alleehol.mpe.mpg.de) left irc: Quit: Konversation terminated! |
| allee-k (~allee-k@alleehol.mpe.mpg.de) joined #fai. |
| 13:21 mgoetze | well, i've yet to find a version of live-initramfs with which i can use 2.6.33 :( |
| 13:24 Mrfai | mikap: mail ist unterwegs. |
| 13:25 Mrfai | mgoetze: ich denke das sind zwei Probleme. 1) 2.6.33 hat auf deiner Hardware kernel panic. 2) 2.6.33 hat kein /var/lib/fai/config in Zusammenhang mit bestimmten live-initrmfs versionen. |
| 13:26 mgoetze | Mrfai: aber mit der neueren live-initramfs version komm ich doch weiter als wo der kernel panic mit den aelteren live-initramfs versionen auftritt |
| 13:27 mgoetze | Mrfai: also wenn kernel panic, dann bevor fai ueberhaupt gestartet ist und versucht configspace zu mounten |
| 13:31 Joelio | any tips for using git to save config space? looking at using a VCS for config, currently it's just on the same server |
| simNIX (~simNIX@156-60.bbned.dsl.internl.net) joined #fai. |
| allee-k (~allee-k@alleehol.mpe.mpg.de) left irc: Quit: Konversation terminated! |
| allee-k (~allee-k@alleehol.mpe.mpg.de) joined #fai. |
| 14:15 allee | mikap: Sun X2100 M2 and X4100 M2 are booting fine with lucid kernel. There's a long delay during/after network setup but installing the demo config succeeds |
| Action: allee tries Dell R410 more recent hardware that the suns ... |
| 14:32 mikap | allee: ok, strange |
| 14:33 mikap | mgoetze: Mrfai: thx, i'm wondering why it works with old l-i version and don't have any idea what's going wrong on your systems |
| Action: mgoetze goes to see whether the r210 will at least work with 2.6.31 |
| 14:45 Mrfai | mikap: currently I'm debugging using break=live-bottom |
| 14:46 mikap | Mrfai: thanks |
| 14:48 mgoetze | nope, kernel panic too... what l-i version did i build my 2.6.28 with? hm |
| 14:49 Mrfai | mikap: VERY STRANGE |
| 14:49 Mrfai | using live-initramfs 1.177... which does not work |
| 14:49 Mrfai | break=live-bottom |
| 14:49 Mrfai | givbes me a shell |
| 14:49 mikap | ok |
| 14:49 Mrfai | typing exit, it continues and fails |
| 14:49 mikap | Mrfai: that's what we expect, no? |
| 14:49 Mrfai | same but with ls /root/var/lib, then exit works |
| 14:50 Mrfai | sorry ls /root/var/lib/fai |
| 14:50 Mrfai | then exit |
| 14:50 Mrfai | I did it twice, because it was unbeliveable |
| 14:51 Mrfai | Seems like a strang aufs bug |
| 14:51 mikap | Mrfai: uff |
| 14:52 mgoetze | hm, so 2.6.28 with 1.157.4-1grml.01 does not produce a kernel panic, but 2.6.33 with 1.157.4-1grml.01 does :( |
| 14:54 mgoetze | does the current squeeze kernel include aufs support? |
| 14:55 mikap | Mrfai: what does the FAI part exactly do for the nfsroot stuff? mounting aufs on aufs? |
| 15:00 Mrfai | FAI does nothing special concerning aufs. It's only using live-initramfs and sets root=/dev/nfs nfsroot=/srv/fai/nfsroot boot=live in pxelinux.cfg |
| 15:00 mikap | hm |
| 15:02 Mrfai | maby this part in scripts/live causes problems: mount -o move /cow /root/live/cow. But it's the same in both live versions |
| 15:03 mikap | i just don't get why it works for you with older l-i |
| 15:13 Mrfai | mikap: I diffed live.log of bad and ok boot process using l-i 1.177... |
| 15:13 Mrfai | nothing important. only a proccess id |
| 15:13 mikap | Mrfai: jepp :-/ |
| 15:14 Mrfai | and a newline |
| 15:14 Mrfai | maybe the old l-i accesses /root/var/lib/.. in some way, so it works |
| 15:15 mikap | hm |
| 15:24 Mrfai | It also works if I do this at the initramfs promt: ls /root/* |
| 15:24 Mrfai | It seems that triggering a readdir helps. |
| 15:25 mikap | hmmmm |
| 15:26 mikap | the ls /root/* >/dev/null 2>&1 would be an ugly workaro^W^Wsolution though i'd prefer to find the cause of the problem |
| 15:29 Mrfai | cause of the problem : buggy aufs |
| 15:30 Mrfai | That's why ubuntu is using unionfs-fuse |
| 15:31 mikap | unionfs-fuse is slooooooooow |
| 15:32 mgoetze | maybe we need a grml2ram-like option for fai :) |
| 15:41 Mrfai | mikap: even a ls /root/var/lib without executing the ls but hitting the tab key (showing a list of a subdirs) works. |
| raphink (~quassel@193.252.149.222) left irc: Remote host closed the connection |
| siggg (~jean@tra78-3-82-244-85-196.fbx.proxad.net) left irc: Quit: Leaving. |
| raphink (~quassel@193.252.149.222) joined #fai. |
| 15:48 Mrfai | doing a ls /live/image/live/filesystem.dir/var/lib also works |
| 15:48 Mrfai | So, IMO it's not aufs allone, it's maybe also nfs caching. |
| 15:49 Mrfai | Without accessing the directory, aufs does not know which subdirectories are available |
| 15:50 Mrfai | that are different from the directory structure that was available inside the initrd |
| 15:58 mgoetze | hm, my colleagues are using 2.6.31-grml with l-i from lenny and it works on the r210 |
| Action: mgoetze tries to build 2.6.33-grml64 with l-i 1.156.1+1.157.2-1 |
| Mrfai (~lange@134.95.9.136) left irc: Quit: leaving |
| raphink (~quassel@193.252.149.222) left irc: Remote host closed the connection |
| edrz (~eric@static-71-178-198-5.washdc.fios.verizon.net) joined #fai. |
| 17:15 mgoetze | mikap: ok, it seems my kernel panics were due to too old firmware-bnx2 |
| 17:16 mgoetze | mikap: however, with 2.6.33-grml64 and l-i 1.173.3-1 i also have the /var/lib/fai/config doesn't exist problem |
| 17:31 Joelio | Hi, I'm having a little issue trying to get this working as a script.. http://paste.debian.net/66805/ |
| 17:32 Joelio | I've tried adding $ROOTCMD, but it won't got to the $target/etc directory first |
| 17:43 mikap | mgoetze: i think it's a readdir() issue |
| allee-k (~allee-k@alleehol.mpe.mpg.de) left irc: Ping timeout: 480 seconds |
| 17:52 mgoetze | Joelio: $ROOTCMD starts a subshell, so your current working directory is forgotten at the end of the $ROOTCMD line |
| 17:52 mgoetze | Joelio: you'll have to do $ROOTCMD (foo; bar; baz) or something like that |
| 17:55 mikap | mgoetze: http://grml.org/live-initramfs_1.177.1-1grml.02_all.deb - could you please give it a shot? |
| 17:56 mgoetze | mikap: sorry, i'm not at work anymore |
| 17:56 mikap | mgoetze: ok, what's the fastest way to reproduce it for me? :) [i've fai installed but no nfsroot ;)] |
| 17:58 mgoetze | mikap: ok, in /etc/fai/apt configure sources and preferences so your kernel and initrd will be pulled |
| 17:59 mgoetze | mikap: in /etc/fai/NFSROOT add linux-image-2.6.33-grml64 |
| 17:59 mikap | ok |
| 17:59 mgoetze | mikap: then run make-fai-nfsroot, it should write stuff to /srv/fai/nfsroot by default |
| 17:59 mikap | ok |
| 18:00 mgoetze | mikap: for netboot you'll need a dhcp+tftp infrastructure of course :) |
| 18:00 mikap | mgoetze: how much space do i need for the default nfsroot ca.? |
| 18:00 mikap | mgoetze: have that basically for grml-terminalserver already :) |
| 18:01 mgoetze | mikap: my nfsroot is 616MB |
| 18:01 mikap | mgoetze: thx |
| allee-k (~allee-k@alleehol.mpe.mpg.de) joined #fai. |
| 18:01 mikap | mount -t tmpfs -o size=800M none /srv/ :) |
| 18:01 mikap | let's speed up the buddy |
| 18:01 mgoetze | hehe |
| 18:02 mgoetze | well it's tolerable even writing to disk in a xen domu, so... |
| 18:03 mikap | i work with tmpfs quite often, also for grml-live, cowbuilder,& |
| 18:04 mgoetze | on my home desktop /var/run, /var/lock and /tmp are all tmpfs |
| 18:07 mikap | mgoetze: rerunning make-fai-nfsroot just updates the nfsroot? |
| 18:07 mgoetze | mikap: yes, and tries to copy the kernel and initrd to /srv/tftp/fai (by default) |
| 18:08 mikap | mgoetze: thx |
| 18:08 mgoetze | mikap: well, updates is maybe the wrong word... more like "demolishs and rebuilds" :) |
| ErKa (~keryell@APuteaux-154-1-34-59.w83-199.abo.wanadoo.fr) joined #fai. |
| 18:09 mikap | mgoetze: that's something i never understood why it should be necessary by default ;) |
| 18:11 mgoetze | well, it guarantees reproducibility... so i don't need to worry about backing up my nfsroot etc., as long as i have my /etc/fai safe :) |
| 18:11 mikap | mgoetze: hehe |
| allee-k (~allee-k@alleehol.mpe.mpg.de) left irc: Ping timeout: 480 seconds |
| simNIX (~simNIX@156-60.bbned.dsl.internl.net) left irc: Remote host closed the connection |
| 18:25 mikap | mgoetze: what's the recommended way to set custom classes for make-fai-nfsroot so i can install a base.tgz in /etc/fai/config/basefiles/? |
| 18:27 mikap | "Create the FAI nfsroot directory as pointed by $NFSROOT in the config_file. [...] The config_file location is discussed in the OPTIONS below." and "-f config_file Not supported any more. Use option -C instead." hmmmm ;) |
| 18:32 mgoetze | mikap: hm classes aren't really used during make-fai-nfsroot, just the arch stuff |
| 18:32 mgoetze | mikap: in my setup i just have multiple fai config dirs |
| 18:32 mikap | mgoetze: hm, so i can't provide a base.tgz? |
| 18:33 mgoetze | mikap: you want to spare yourself the debootstrap? i don't think m-f-n can do that :) |
| 18:33 mikap | mgoetze: sure, that would speed up the operation a lot |
| 18:33 mikap | mgoetze: pfuh, i hate myself for having to contribute to every single bit of software whenever i'm touching it ;) |
| 18:36 mgoetze | mikap: heh :) |
| ErKa (~keryell@APuteaux-154-1-34-59.w83-199.abo.wanadoo.fr) left irc: Ping timeout: 480 seconds |
| 18:45 mikap | mgoetze: implemented the feature already ;) |
| allee (~ach@allee.mpe.mpg.de) left irc: Remote host closed the connection |
| 18:48 mikap | /srv/fai/nfsroot/live/filesystem.dir already exists. Removing /srv/fai/nfsroot/live/filesystem.dir |
| 18:48 mikap | Extracting /srv/base.tgz: done |
| 18:48 mikap | through running make-fai-nfsroot -v -B /srv/base.tgz |
| mfl (~mfl@adonis.fnb.maschinenbau.tu-darmstadt.de) left irc: Quit: leaving |
| 18:52 mikap | Errors were encountered while processing: /var/cache/apt/archives/fai-nfsroot_3.2.17~lenny1_all.deb - hm |
| 18:54 mikap | ah, i've to create /.THIS_IS_THE_FAI_NFSROOT |
| 19:05 mikap | http://grml.org/tmp/support_base_tgz_in_make-fai-nfsroot.patch |
| mfl (~mfl@adonis.fnb.maschinenbau.tu-darmstadt.de) joined #fai. |
| 20:02 mikap | mgoetze: could it be that FAI_ACTION=sysinfo doesn't trigger the problem or does it require a valid fai config space? |
| 20:24 mikap | mgoetze: http://grml.org/screeni/gkrellShoot_10-03-31_222329.png - does this look OK? |
| 20:51 mgoetze | mikap: dunno, i've never used sysinfo :) |
| 20:51 mikap | mgoetze: any fast options to test it? :) |
| 20:52 mgoetze | mikap: your screenshot looks like a success |
| 20:52 mgoetze | i'll confirm it tomorrow |
| 20:53 mikap | mgoetze: thx! |
| siggg (~jean@tra78-5-88-172-128-32.fbx.proxad.net) joined #fai. |
| siggg (~jean@tra78-5-88-172-128-32.fbx.proxad.net) left irc: Quit: Leaving. |
| --- Thu Apr 1 2010 |
