The following are reported as incompatible with models other than the 
original Atari 400/800.  Many can nevertheless be made to run on XL/XE's
if you use a translator to run the original 400/800 OS on your XL/XE. 
 
Apple Panic                    Broderbund 
Aquatron                       Sierra On-Line 
Astro Chase                    (by First Star Software) Parker Bros.
Atari Word Processor           Atari
Atlantis (early only?)         Imagic (at least most copies OK on XL/XE)
Attack at EP-CYG-4             (by Bram) Romox
Bacterion!                     Kyle Peacock/Tom Hudson/ANALOG#20
   patch for XL/XE available:
   http://www.cyberroach.com/analog/an20/bacterion_patch.htm
Bandits                        Sirius Software
BearJam                        Chalk Board
Chicken                        Synapse
Crossfire                      Sierra On-Line (keyboard doesn't work on XL/XE)
Dancing Feats                  (by Softsync) Romox
Demon Attack                   Imagic (Activision re-release fixed for XL/XE)
Disk 50                        Star Soft International (SSI)
Dreadnaught Factor, The        Activision 
Drelbs                         Synapse 
File Manager 800+              Synapse
Forbidden Forest (early only?) Cosmi (at least most copies OK on XL/XE)
Fort Apocalypse -- cartridge   Synapse (all disk/tape releases OK on XL/XE)
Galahad And The Holy Grail     APX 
Go                             Hayden 
Gorf                           Roklan
Jawbreaker II                  Sierra On-Line
Jet Boot Jack (early only?)    English Software (at least most OK on XL/XE)
Juggler                        IDSI 
K-Razy Antiks                  (by Kay Enterprises for K-Byte) CBS 
K-Razy Kritters                (by Kay Enterprises for K-Byte) CBS
K-Razy Kritters                (by Kay Enterprises) K-Byte
K-Star Patrol                  (by Kay Enterprises for K-Byte) CBS
Kangaroo (prototype)           (Atari) 
KoalaPainter                   Koala 
Leo's 'Lectric Paintbrush      Chalk Board
Leo's Links                    Chalk Board 
Letter Perfect (before v6)     LJK (all version 6.x releases OK on XL/XE)
LogicMaster                    Chalk Board 
Mac/65 [ver. 1.00, orange]     OSS (all releases after 1.00 OK on XL/XE)
Maze                           Epyx 
Micro Illustrator              Chalk Board
MicroMaestro                   Chalk Board 
Monkey Wrench                  Eastern House
Monster Maze                   Epyx 
Ms. Pac-Man                    Atari (glitches with late-production XE units)
M.U.L.E. (early only?)         (by Ozark Softscape) Electronic Arts 
                               (most copies/releases OK on XL/XE)
Nautilus                       Synapse
  XL/XE workaround: hold down START to skip the title screen,
  which is where it locks up. --Scott Stilphen, 6 Jun 2007
Picnic Paranoia                Synapse 
Pool 1.5                       IDSI 
Pool 400                       IDSI 
Protector II                   Synapse 
QS Forth                       James Abanese / [QS] Quality Software
Rack 'Em Up                    Rocklan 
Shamus                         Synapse 
  XL/XE workaround: hold down START or SELECT to skip the title screen,
  which is where it locks up.
  A re-release by Americana/Synsoft corrects the incompatibility problem.
  (http://www.atarimania.com/game-atari-400-800-xl-xe-shamus_6174.html)
  --Scott Stilphen, 6 Jun 2007
Slime                          Synapse 
Snapper                        Silicon Valley Systems 
Space Dungeon                  Atari 
Squish 'Em                     Sirius
Story Machine -- cartridge     Spinnaker (disk release OK on XL/XE)
Synassembler                   Synapse
Text Wizard                    Datasoft
Zaxxon (early release?)        Datasoft (most copies/releases OK on XL/XE)

Konrad M.Kokoszkiewicz writes: 
 
XL/XE software won't work on 400/800 if: 
 
1) it uses shadow RAM at $C000-$CFFF and $D800-$FFFF 
2) it uses RAM expansions at $4000-$7FFF controlled by PORTB $D301 
3) it uses specific XL OS functions (like JNEWDEVC) 
4) it uses illegal XL OS addresses. 
5) it uses European Charset :) 
 
Andreas Koch adds:

To get an overview or see a chart of OS changes from the 800 to the XL
line, refer to Antic magazine Volume 3, Number 2 (June 1984), pages 10-14;
(online: http://www.atarimagazines.com/v3n2/insideatari.html )
Also note, that some software will not work correct (or not at all) on
newer XE/XEGS versions (which have a new OS with a new version number, 
a new Self Test/Memory Test/Keyboard Test, larger RAM chips, etc. etc.);

Thomas Richter contributes further details (16 Jan 2004):

There are a couple of reasons why some games don't run on the XL/XE
models.  I try to order them by "likeliness", of course biased by my
personal observations:

i) The printer buffer of the XL Operating System in page 3 is a couple
of bytes shorter.  The additional bytes are used for extended OS
variables not available in the 800 series.  Most prominent is the $3fa
location, holding a shadow register of GTIA's TRIG3 signal.  While a
true joystick trigger line in the 400/800 series, this signal is used
as "cart inserted" signal for XL/XE models.  Unfortunately, the OS
compares GTIA trig3 with the shadow register at $3fa in each vertical
blank, running into an endless loop if the register contents don't
match.  This causes hangs for games using page 3 either as copy-buffer
or for player-missile graphics.  (Hangs by Ms. Pac-Man and
Bacterion! are caused by this, and many others...)  This is "fixable"
either by the translator disk, or by a quick hack into the game,
replacing the OS vertical blank or poking TRIG3 frequently into its
shadow.  The reason for the OS behavior might be that Atari wanted to
prevent crashes if the cartridge is inserted or removed while the
machine is running.  The 400/800 is powered down when a cart is
inserted, the XL/XE lacks the cover of the older models that triggered
a little switch to interrupt the power line.

ii) Similar to the above, writes to $3f8.  This OS equate defines
whether on a warm start, the BASIC ROM shall be mapped back in.  If
its contents are altered, a program triggering a reset as part of its
initialization will find itself then with 8K less RAM occupied by
a BASIC ROM, making it crash.  Similarly, writes to the cartridge checksum
$3eb could cause a cold-start on a "reset initialization".  This is
fixable by the translator disk.

iii) Some games use a four-joystick setup, or at least initialize
PIA itself.  If this happens inadequately, PIA Port B, bit 0 gets changed,
disabling the ROM, and thus crashing the machine.  This is not fixable
by the translator since it is a hardware issue.

iv) Direct jumps into the OS ROM, not using the documented vectors in
the $e450 area.  Interestingly, this fault is not as common as it may
sound since games hardly ever use the OS.  It causes failures of
some "serious applications", most notably "QS Forth" and applications
compiled by it.  This is fixable by the translator disk.

As a side remark, it is interesting to note that no such documented
jump-ins exist for the math-pack ($d800 to $dfff).  It is not really
part of the OS, but looks more like a part of the BASIC interpreter
that didn't make it into the OS because there was no room left.  Thus,
direct jump-ins have to be used here that are documented in the De Re
Atari (for example).  Atari never changed them, but it seems likely
that this documentation happened more or less as an accident since the
same source also lists some mathematics-related jump-ins into the
Basic (namely, to compute SIN and COS and related) that are only valid
for the Rev. A BASIC.  Thus, the math pack might be a couple of
routines that have been originally intended for "private use" of the
BASIC ROM, but then have been found "too useful" by many others to
remain "closed".  Otherwise, it is hard to explain why the otherwise
pretty cleaned-up OS comes with a construction like this.