After the move to the E5-2670 CPU's I'm seeing illegal instruction errors coming out of some programs.

I was happily using -march=native on the old Xeon's.

I didn't rebuild the entire build toolchain, only gcc, but even with something like -march=generic I'm still getting errors.

Anyone have successful CFLAGS and advice on whether I should rebuild entire toolchain?

I've been using -march=nocona for years with no problems, but I've only been upgraded to E5-2630L so not sure if that matters.

I'd think if you had a problem like that you'd need to rebuild world, not just the toolchain. At the least you'd want to figure out which programs are causing the illegal instructions and rebuild their entire dependency tree.

If you use -march=native, you may need to add -mno-avx to CFLAGS, then start rebuilding stuff. AFAIK AVX is not enabled on Linodes (upstream Xen issue?).

Yikes, can someone confirm this?

I was just about to rebuild (I use gentoo as well) a few packages explicitly taking advantage of the AVX now that I'm no longer on an older architecture.

It is definitely disabled. Please check for the absence of the AVX extention in your /proc/cpuinfo to confirm . (pretty sure they haven't enabled it...)

It's definitely disabled. Attempting to use AVX results in illegal instructions, which you can prove quickly and easily by building xz-utils with --march=native, and then try to use it. When disassembled at the point where the illegal instruction is thrown, you'll find that gcc decided it would be a good idea to use an AVX instruction while calling a function. Retrograde's suggestion will work, as will using an --march setting which doesn't include AVX settings; the highest of these being corei7. If you don't know of anything that can take advantage of the PCLMUL instructions, then you can just stick with corei7. The real bug here lies with gcc, and one of these days I'll get around to poking them about it. You'll see this on all hosts with E5 series processors, as they're all the same design, just slightly different clock speeds and cache sizes.

_________________
Disclaimer: I am no longer employed by Linode; opinions are my own alone.

Did some digging, and I'm currently rebuilding using this:

yurizoku ~ # cat /etc/portage/make.conf
CFLAGS="-Os -march=corei7-avx -mno-avx"
CXXFLAGS="${CFLAGS}"
USE="mmx sse sse2 ipv6 vhosts -gpm -sendmail -X threadsafe threads"
MAKEOPTS="-j12  --load-average=7.75"
EMERGE_DEFAULT_OPTS="--jobs 3 --load-average=7.75"
PYTHON_ABIS="2.7"
PYTHON_TARGETS="python2_7"
PYTHON_SINGLE_TARGET="python2_7"
ACCEPT_KEYWORDS="x86"

So far it seems to be right for a host I'm on which shows this in CPUinfo:

yurizoku ~ # cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz

...

flags           : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ss ht nx constant_tsc nonstop_tsc pni pclmulqdq ssse3 sse4_1 sse4_2 popcnt tsc_deadline_timer aes hypervisor ida arat epb pln pts dtherm

Apparently it's not the same to do

-march=corei7-avx -mno-avx

versus

-march=corei7

This is because there are apparently more differences than just the (not currently working on linode for domU guests) avx instructions, so I opted for the former set of cflags since it more closely matches the architecture & only disables AVX-related instructions & optimizations; Doing it this way will still be right for the CPU on my host, but falls back onto using a velvety smooth paste made unicorn tears, pixie dust, and/or makes full use of all available non-avx instructions & CPU features for accelerating (insert relevant list of things) whenever the compiler optimizations determine it is appropriate

as for the other thing you may have noticed in my cflags, I prefer -Os instead of -O2 or -O3 because gentoo officially recommends against -O3 due to occasionally breaking packages in unexpected ways, and I'm using the -Os instead of gentoo's usual preferred -O2 since it produces smaller binaries than -O2... Not sure how important it is to others, but I like it because I feel there's a better chance for my binaries fit in the cache better (which may or may not make a difference, considering there are probably other guests on my host... YMMV)

Edited to add:

kuzetsa@yurizoku ~ $ cat ~/now && ~/now && head -n 1 /etc/portage/make.conf     #!/bin/bash
# kuzetsa's timestamp generator


date +%A_%B_%d_%Y_%H%M%S
cat /proc/version
uptime
Friday_October_11_2013_084300
Linux version 3.10.3-x86-linode53 (root@build) (gcc version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Jul 26 12:25:24 EDT 2013
 08:43:00 up  1:53,  2 users,  load average: 2.57, 3.73, 3.68
CFLAGS="-Os -march=corei7-avx -mno-avx"

It's working so far... I've got gnu screen installed to keep my 'emerge -e world' running in the background while I rebuild everything using these CFLAGS:

CFLAGS="-Os -march=corei7-avx -mno-avx"

Guess I can confirm it works on my host... as you can see, it's a host with the new E5-2670 CPUs:

kuzetsa@yurizoku ~ $ uname -a
Linux yurizoku 3.10.3-x86-linode53 #1 SMP Fri Jul 26 12:25:24 EDT 2013 i686 Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz GenuineIntel GNU/Linux

Nothing has broken so far, and I'm planning on updating kernels over the weekend. I'll be staying on the 3.10.x LTS linux kernel branch since it seems to work fine on this hardware (and since it is now 4th quarter 2013, I will be use "something other than debian's ancient gcc 4.4.5 which hasn't been current since 4th quarter 2010")

Edit 2:

After building GCC with these flags, rebuilding all packages from source, and rebuilding kernel:

CFLAGS="-Os -march=corei7-avx -mno-avx"

I can now say these flags are confirmed, and have work flawlessly for over 24 hours on my E5-2670 linode host (like I said, gentoo technically recommends -O2 but it's just a matter of preference... the -Os version just skips a few optimizations which increase the size of the compiled binary)