Having seen both films but not read the books, I can only comment on the film versions. The Big Sleep is a definite favorite movie of mine, with excellent acting, nice plotting and a great use of withheld information leading to a very exciting twist-and-turn story dynamic. This has got to be the best film that William Faulkner ever worked on in Hollywood and, looking at the names involved on this project, that makes sense.
Raymond Carver, William Faulkner, Humphrey Bogart, Lauren Bacall and Howard Hawkes all bring/brought something creatively significant to the table for this film.
Along with Chinatown, I think The Big Sleep is one of the greatest detective noir films ever made.
The Maltese Falcon boasts an impressive list of names on the project too, with Bogart also, but the visual style is a bit more stodgy in my opinion. And the heavy reliance on symbolism (both narrative and visual) might ultimately detract from the power of the story instead of add to it.
These movies are so similar that we might compare them in this simple way - if you had to choose one to keep in your collection and toss the other, which would you choose to keep?