Though the ESB replacement concept is the same for most SOA Gateways, vendor implementations can be very different. Each vendor can provide a laundry list of their supported protocols and message formats. Each will take a unique approach to policy configuration and extensibility. Each will support slightly different encryption algorithms and access control mechanisms. Each will have various potential deployment methods and clustering strategies.
When considering an SOA Gateway in an ESB scenario, the goal is to match up each of these capabilities with your particular requirements. Consider which of your current applications could be made more valuable by providing a robust, standards-based interface. Consider what data you'd like to expose, to whom and in what format. Then go through the various options and choose wisely. If you want to provide an OAuth-protected REST interface to a mainframe application accessed using MQ, then make sure the necessary formats and protocols are supported by your SOA Gateway vendor.
Support for these common ESB functions is what will likely make your decision much easier. In terms of protocols, what do you need beyond basic HTTP(S)? Are messaging protocols (MQ, JMS, EMS) a requirement; if so, do you use a particular flavor (vendor-specific JMS), and is it supported? Do you need file-based protocols such as FTP and NFS; if so, which secure versions and/or security options (FTPS, SFTP, NFSv4)?
Do you need support for incoming (POP3, IMAP) or outgoing (SMTP) email? In terms of message formats, are XML-based options (XML, SOAP) sufficient? Do you need flat-file support? B2B formats such as EDI? Mainframe formats like Cobol Copybooks? Modern Web-based formats such as JSON?
What tools will you use to map between these formats? How is policy created on the gateway, using what interfaces? Is there a logical GUI that makes it clear what actions are taking place? Is there a Command Line Interface? How about a services-based interface for programmatic access to gateway functions?
On the security side, what incoming and outgoing credential types need to be supported, and is there an easy, standards-based way of mapping between them? What authentication and authorization servers are supported? Does the gateway both support modern cryptographic algorithms and protect against modern threats? Do you require any security certifications or specialized hardware?