O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

SGI hardware problems, solutions, tips, hacks, etc.
Forum rules
Any posts concerning pirated software or offering to buy/sell/trade commercial software are subject to removal.
User avatar
ShadeOfBlue
Moderator
Moderator
Posts: 799
Joined: Tue Nov 25, 2003 12:09 pm
Location: Europe

O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

Unread postby ShadeOfBlue » Tue Oct 28, 2008 11:19 am

The newest L2 firmware is broken -- upgrading to it will result in loss of L2 network (telnet) & upgrade functionality!

It is still possible to connect to the L2 controller via the serial console & use the LCD normally (also, the corresponding L1 firmware version doesn't have any problems).

After looking around in L2's 'shell' mode, I've traced the problems to a missing shared library (libscan.ppclinux.so) in the firmware image:

Code: Select all

/stand/sysco/bin/l2flash: error in loading shared libraries: libscan.ppclinux.so: cannot open shared object file: No such file or directory

The networking stuff has the same dependency and since the flash utility never starts, it is now almost impossible to re-flash it with an older firmware (I even tried the '-l2recover' mode in flashsc).

Way to go, SGI...


I would be very grateful if someone could help me fix this.

It should be possible to get the missing file from an L2 that has it (assuming it is present in older firmware versions).
Since the L2 doesn't have USB mass storage support, that would have to be done like this:
  • make sure your terminal's line history size is set to unlimited
  • telnet into the L2 (or connect via the serial port)
  • enter shell mode by typing '!' & pressing enter
  • check if the file exists by doing a 'ls -l /stand/sysco/lib/'
  • then do a 'cat /stand/sysco/lib/libscan.ppclinux.so' (at this point, the terminal will display garbage and switch into bold or underlined mode)
  • wait for the output to stop and the prompt to appear, then leave shell mode with 'exit'
  • disconnect from the L2
  • save the output of the session to a file (most terminals offer a way of doing this)
  • please post the output file here as an attachment

Then I will extract the .so file from the output, do a 'cat > libscan.ppclinux.so' (in the /stand/sysco/lib/ directory) on my L2 and feed it the file.
After that, things should start working :)
[OK, I can think of at least two ways this could fail, but it's worth a shot]

Any other suggestions are welcome.

User avatar
nekonoko
Site Admin
Site Admin
Posts: 8145
Joined: Thu Jan 23, 2003 1:31 am
Location: Pleasanton, California
Contact:

Re: O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

Unread postby nekonoko » Tue Oct 28, 2008 11:34 am

Okay, here's the output as requested. My L2 version is 1.48.0.
Attachments
Local 10-28-08.txt
(177.67 KiB) Downloaded 48 times
Twitter: @neko_no_ko
IRIX Release 4.0.5 IP12 Version 06151813 System V
Copyright 1987-1992 Silicon Graphics, Inc.
All Rights Reserved.

User avatar
ShadeOfBlue
Moderator
Moderator
Posts: 799
Joined: Tue Nov 25, 2003 12:09 pm
Location: Europe

Re: O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

Unread postby ShadeOfBlue » Tue Oct 28, 2008 1:33 pm

Thank you!

I've checked the output and I see that your terminal program probably converted the linefeed characters, as the size of the file is about 20kB less than that in the 'ls -l' output :(

After some more digging in the L2 firmware now, I noticed it has an FTP server running through inetd! I can't test it, since networking on my L2 is down, but if it has the entire filesystem exported, it should be possible to get the file this way.

There's also a rudimentary tftp client, but that requires a TFTP server running on another machine to be able to do something like 'tftp put /stand/sysco/lib/libscan.ppclinux.so <server_IP>:/tftp/libscan.ppclinux.so'.

Sorry for wasting your time with the terminal output, I should have checked for alternatives first.

User avatar
recondas
Moderator
Moderator
Posts: 5441
Joined: Sun Jun 06, 2004 5:55 pm
Location: NC - USA

Re: O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

Unread postby recondas » Tue Oct 28, 2008 2:26 pm

If it's in any way helpful to you, I have a slightly older version:

Code: Select all

Current L2 version 1.12.1


I also wonder if it's possible to downgrade your firmware.

There's certainly an element of risk involved, but there's at least a chance you could recover in the event of a failed downgrade. The the O3k manual mentions recovery from a failed L2 flash:
the manual wrote:If the L2 update fails, there is no second image as a back up as there is with the L1. The L2, however, will not run the kernel image if it is not valid. The L2 is intelligent enough at this point to be upgraded through its console port, as follows:

Code: Select all

$> /usr/cpu/firmware/sysco/flashsc --l2recover /usr/cpu/firmware/sysco /l2.bin <device>
If it is possible the next question would be sourcing an older version of the l2.bin file. If they were distributed along with IRIX revisions, it might be possible to extract one using bri3d's .sw/.idb distribution image format extraction trick.
***********************************************************************
Welcome to ARMLand - 0/0x0d00
running...(sherwood-root 0607201829)
* InfiniteReality/Reality Software, IRIX 6.5 Release *
***********************************************************************

User avatar
nekonoko
Site Admin
Site Admin
Posts: 8145
Joined: Thu Jan 23, 2003 1:31 am
Location: Pleasanton, California
Contact:

Re: O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

Unread postby nekonoko » Tue Oct 28, 2008 6:14 pm

ShadeOfBlue wrote:After some more digging in the L2 firmware now, I noticed it has an FTP server running through inetd! I can't test it, since networking on my L2 is down, but if it has the entire filesystem exported, it should be possible to get the file this way.


Ack - well, I don't have networking set up on my L2 and don't really have the spare time to muck with setting that up. If it had something like an xmodem implementation that would be great - I could just do a serial file transfer. Hopefully somebody else has a networked L2.
Twitter: @neko_no_ko
IRIX Release 4.0.5 IP12 Version 06151813 System V
Copyright 1987-1992 Silicon Graphics, Inc.
All Rights Reserved.

User avatar
Zerolapse
Posts: 418
Joined: Mon Sep 12, 2005 8:32 pm
Location: New York, USA

Re: O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

Unread postby Zerolapse » Wed Oct 29, 2008 7:06 am

The FTP daemon does indeed appear to export the entire L2 filesystem, although I'm running version 1.32.4, so I'm not sure if that would be helpful for you.

-Jesse
:O3x06R: :ChallengeL: :Octane: :Octane2: :O2: :Indy: (4x Challenge S)
The current fleet.

User avatar
ShadeOfBlue
Moderator
Moderator
Posts: 799
Joined: Tue Nov 25, 2003 12:09 pm
Location: Europe

Re: O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

Unread postby ShadeOfBlue » Wed Oct 29, 2008 7:16 am

recondas wrote:If it's in any way helpful to you, I have a slightly older version:

Code: Select all

Current L2 version 1.12.1

There's a chance it will have missing symbols, but it's worth a try :)

recondas wrote:I also wonder if it's possible to downgrade your firmware.

I've already tried this -- the recover option returns error 8, whatever that is. I think it either needs a running l2flash program on the L2 or it runs from its bootloader when it can't run the kernel.

recondas wrote:If it is possible the next question would be sourcing an older version of the l2.bin file. If they were distributed along with IRIX revisions, it might be possible to extract one using bri3d's .sw/.idb distribution image format extraction trick.

I've extracted the L2 image from the 6.5.15 overlay CD's (that utility worked great for this :)), except I can't do anything with it now... I've been thinking of flashing the L2 firmware by hand, but that could render the entire L2 inoperable if I screw up.

Here's the L2 firmware I extracted in case anyone needs it to recover a really broken L2 (I hope it's legal to post it here):
l2.bin
L2 firmware from IRIX 6.5.15
(2.73 MiB) Downloaded 37 times

I don't know exactly what version it is, but I guess it is quite old.


nekonoko wrote:Ack - well, I don't have networking set up on my L2 and don't really have the spare time to muck with setting that up. If it had something like an xmodem implementation that would be great - I could just do a serial file transfer. Hopefully somebody else has a networked L2.

It doesn't have xmodem or uuencode, but I found 'od' -- 'od -t x1 /stand/sysco/lib/libscan.ppclinux.so' should make a hex dump of the file :)
I can then write a program to convert the dump back into a file.


Zerolapse wrote:The FTP daemon does indeed appear to export the entire L2 filesystem, although I'm running version 1.32.4, so I'm not sure if that would be helpful for you.

As I mentioned above, it's possible the library will have missing symbols and won't run, but it would still be worth to try it.


Thanks to all of you for your assistance.

User avatar
recondas
Moderator
Moderator
Posts: 5441
Joined: Sun Jun 06, 2004 5:55 pm
Location: NC - USA

Re: O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

Unread postby recondas » Wed Oct 29, 2008 9:16 am

ShadeOfBlue wrote:
recondas wrote:If it's in any way helpful to you, I have a slightly older version:

Code: Select all

Current L2 version 1.12.1

There's a chance it will have missing symbols, but it's worth a try :)

I haven't configured networking either - it's something that up until now was way down on my to-do list. It'll probably be the weekend before I have the time and opportunity though.

ShadeOfBlue wrote:It doesn't have xmodem or uuencode, but I found 'od' -- 'od -t x1 /stand/sysco/lib/libscan.ppclinux.so' should make a hex dump of the file :) I can then write a program to convert the dump back into a file.
When I get the L2 networked, I'll try this.
***********************************************************************
Welcome to ARMLand - 0/0x0d00
running...(sherwood-root 0607201829)
* InfiniteReality/Reality Software, IRIX 6.5 Release *
***********************************************************************

User avatar
nekonoko
Site Admin
Site Admin
Posts: 8145
Joined: Thu Jan 23, 2003 1:31 am
Location: Pleasanton, California
Contact:

Re: O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

Unread postby nekonoko » Thu Oct 30, 2008 1:30 am

ShadeOfBlue wrote:It doesn't have xmodem or uuencode, but I found 'od' -- 'od -t x1 /stand/sysco/lib/libscan.ppclinux.so' should make a hex dump of the file :)
I can then write a program to convert the dump back into a file.


Okay, here's the od dump file:
Attachments
libscan.ppclinux.so.txt
(691.99 KiB) Downloaded 38 times
Twitter: @neko_no_ko
IRIX Release 4.0.5 IP12 Version 06151813 System V
Copyright 1987-1992 Silicon Graphics, Inc.
All Rights Reserved.

User avatar
ShadeOfBlue
Moderator
Moderator
Posts: 799
Joined: Tue Nov 25, 2003 12:09 pm
Location: Europe

Re: O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

Unread postby ShadeOfBlue » Thu Oct 30, 2008 4:30 am

recondas wrote:I haven't configured networking either - it's something that up until now was way down on my to-do list. It'll probably be the weekend before I have the time and opportunity though.

It's really easy -- you just set it a static IP with the 'ip' command from the L2 prompt & reboot (and apparently it can also get an IP via DHCP).

recondas wrote:When I get the L2 networked, I'll try this.

Thank you. I see that nekonoko has posted the output from his L2, so hopefully, this won't be necessary; I do appreciate it, though.


nekonoko wrote:Okay, here's the od dump file:

Excellent, thanks!

Here's the program I wrote to convert it back into a file:
od2file.c
'od -t x1' hexdump to binary file converter source code
(3.47 KiB) Downloaded 46 times

And here's the file itself:
libscan.ppclinux.so.gz
The missing library itself (gzipped)
(53.7 KiB) Downloaded 47 times


Now the only problem I have is that the root filesystem on the L2 (cramfs) is permanently read-only and the only filesystem the L2's kernel supports is cramfs...
*sigh* It's never easy...

I've downloaded the tools necessary to create a cramfs image -- I will put the library in a small image, manually enable networking on the L2, use 'wget' to place it into /tmp (which is writeable) and try mounting it to /stand/sysco/lib/.

The system should then pick up the library and I will finally be able to re-flash it with the older firmware using 'flashsc' from within IRIX.

That's the plan, anyway :)

Thanks again to all who've replied -- I'll post an update later today or tomorrow.

User avatar
nekonoko
Site Admin
Site Admin
Posts: 8145
Joined: Thu Jan 23, 2003 1:31 am
Location: Pleasanton, California
Contact:

Re: O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

Unread postby nekonoko » Thu Oct 30, 2008 4:45 am

ShadeOfBlue wrote:I don't know exactly what version it is, but I guess it is quite old.


I'm pretty sure my current 1.48.0 came from patch 7149 if that helps any. If you need the l2.bin file I can attach it.

Code: Select all

IRIX Patch 7149: System Controller Firmware/Software Updates for IRIX 6.5.24 - 6.5.30    30-Jan-2007


Nice work on that od2file.c program by the way - that could come in handy!
Twitter: @neko_no_ko
IRIX Release 4.0.5 IP12 Version 06151813 System V
Copyright 1987-1992 Silicon Graphics, Inc.
All Rights Reserved.

User avatar
ShadeOfBlue
Moderator
Moderator
Posts: 799
Joined: Tue Nov 25, 2003 12:09 pm
Location: Europe

Re: O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

Unread postby ShadeOfBlue » Thu Oct 30, 2008 11:21 am

nekonoko wrote:If you need the l2.bin file I can attach it.

Please do, it would be nice to have a newer version.

nekonoko wrote:Nice work on that od2file.c program by the way - that could come in handy!

Thanks!

I haven't had the time to test the fix procedure above yet, though, I will do it tomorrow.

User avatar
nekonoko
Site Admin
Site Admin
Posts: 8145
Joined: Thu Jan 23, 2003 1:31 am
Location: Pleasanton, California
Contact:

Re: O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

Unread postby nekonoko » Thu Oct 30, 2008 12:10 pm

Not a problem - the l2.bin file from my system is attached.
Attachments
l2.bin
(2.86 MiB) Downloaded 48 times
Twitter: @neko_no_ko
IRIX Release 4.0.5 IP12 Version 06151813 System V
Copyright 1987-1992 Silicon Graphics, Inc.
All Rights Reserved.

User avatar
ShadeOfBlue
Moderator
Moderator
Posts: 799
Joined: Tue Nov 25, 2003 12:09 pm
Location: Europe

Re: O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

Unread postby ShadeOfBlue » Fri Oct 31, 2008 8:46 am

Thank you.

I was finally able to fix it by manually enabling networking ('ifconfig eth0 192.168.1.17' && 'ifconfig eth0 up') and doing this:

Code: Select all

sh-2.03# mount -t tmpfs -o size=750k /dev/shm /mnt           
sh-2.03# cp /stand/sysco/lib/* /mnt
sh-2.03# cd /mnt
sh-2.03# wget http://192.168.1.13:8080/libscan.ppclinux.so
libscan.ppclinux.so  100% |*****************************|   300     - stalled -
sh-2.03# mount -t tmpfs -o size=750k /dev/shm /stand/sysco/lib
sh-2.03# ls /stand/sysco/lib/
sh-2.03# cp /mnt/* /stand/sysco/lib/
sh-2.03# ls /stand/sysco/lib/
libIRouter.ppclinux.so  libdrivers.ppclinux.so  libscan.ppclinux.so
libRouter.ppclinux.so   libmain.ppclinux.so     libsys.ppclinux.so
libapi.ppclinux.so      libpart.ppclinux.so
sh-2.03# cd /stand/sysco/lib/
sh-2.03# umount /mnt


The l2flash program started on its own after a few minutes:

Code: Select all

INFO: attempting connection to localhost:9002
INFO: Validating connection from 'localhost' (127.0.0.1)
INFO: connection to localhost:9002 established.
INFO: Connection - sgi (sgi) @ localhost running 'l2flash'.


After that, I was able to do a 'flashsc ./l2.bin all' from IRIX with the L2 image provided by nekonoko (thanks!), rebooted the L2 ('l2cmd reboot_l2') and now everything works again! :D

User avatar
nekonoko
Site Admin
Site Admin
Posts: 8145
Joined: Thu Jan 23, 2003 1:31 am
Location: Pleasanton, California
Contact:

Re: O3k users: Don't upgrade L2 controller firmware to 1.44.0 !!

Unread postby nekonoko » Fri Oct 31, 2008 11:09 am

Well that's great news - glad you were able to recover it!
Twitter: @neko_no_ko
IRIX Release 4.0.5 IP12 Version 06151813 System V
Copyright 1987-1992 Silicon Graphics, Inc.
All Rights Reserved.


Return to “SGI: Hardware”

Who is online

Users browsing this forum: No registered users and 3 guests