This site uses advanced css techniques
> I was wondering if any of you has experience in testing fax servers.
I'm the author of one of the oldest  and most popular commercial UNIX fax system (VSI-FAX, now sometimes called Esker Fax), and I can offer some thoughts (in no particular order). In general, this is going to be a hard nut to crack, but it's not inconceivable.
DISCLAIMER: I have not done anything with the ECM (error correcting) modes, nor have I followed the fax spec in the last few years, so new developments may have overtaken my experience. I'm also pretty much limited to class 2 and class 3 fax modems, which handle most of the low-level stuff automatically. Class one modems require host software to bit-bang the T.30 protocol and put a much higher demand on the driver and on the developer. I can talk T.30, but haven't really written to it.
Most fax modems also support data, and with software support, the fax server can also accept a data call and route it through a getty/login process. These arrangements were very popular with smaller offices because the same modem could be used for outbound fax while still being available for inbound data from vendor support personnel. With internet connectivity so common, this is less prevalent, but the modes remain.
Class 2 modems can be told how to answer incoming calls:
When the modem actually answers the call, it tells the host what was ultimately negotiated with the first response. If it's the word CARRIER - possibly followed by a speed or modulation token - then this is a data call, but otherwise the responses are a series of tokens starting with +F. These are fax calls.
By far the best avenue for getting into a system remotely is to find the data mode, because this is a more "traditional" form of entry. In addition to the usual techniques for gaining remote modem access, there may be some additional avenues due to how the fax software hands off the call.
"Login processing" is much harder to get right than simply prompting for a username and a password - there are many fine details required in order to not be fooled by overflows and the like. A responsible fax server will invoke the underlying system's getty or login programs, but a server that tries to "roll its own" processing is more likely to have unfound bugs.
Getting "a regular shell" from a fax-only modem is going to be damn near impossible: the fax protocols are all half duplex. I also believe that once the modem is in fax mode, it's going to be next to impossible to "fool" it into switching into data mode.
Unlike standard data connections, which pretty much just form a conduit from the remote to the local computer (and the modem's command channel is out of the way), a fax conversation is an ongoing set of chitchat between the host and the modem. If the modem gets out of sync - for whatever reason - it's simply going to fail the whole call.
But I can think of three potential avenues, albeit very remote, for finding an overflow when trying to gain access via an inbound fax modem.
I've never done this with security or pen-testing in mind, but there is a lot that can be gathered remotely by paying attention during a fax call. We routinely had to characterize a remote system that was giving us trouble, and these things come to mind.
The order, timing, and format of the data received during the initial part of the call can tell much. The NSF frame(s), if any, can reveal the country and vendor of the unit, and proprietary data inside the NSF frame can sometimes reveal other things (mailbox name, passwords, and the like).
The "DIS" - Digital Information Signal - is particularly useful, as it's a bitmask of many capabilities. Allowable modem rates, data formats, encoding/error control modes, and the like are represented in several dozen bits, and sometimes you can get a hint at what you are looking at.
For instance, finding the minimum scanline time of zero means you're probably talking to a computer, but >0 times mean it's probably an older thermal fax.
I suppose that even before the fax handshake, one could pay attention to the answer tones that fly by before the fax answer tone is heard: the order and timing of these could also reveal what you're calling, though I think you'd need specialized equipment to detect these.
I don't think I've ever heard of an externally exploitable fax server, though I've not researched this since my professional focus turned more toward security.