I write this message as the main author of the Orthanc DICOM server. Please be sure the Orthanc community would be happy to see an integration with OpenEMR.
Several users have reported using Orthanc with its associated PostgreSQL plugin to store around 10 TB of data. So it should not be a problem to use Orthanc to store a database of x-rays images, which looks as your primary goal, given that a typical single RX DICOM image weights roughly about 10MB.
Regarding your answer to Morten, DICOM is conceived as an universal standard to store and exchange medical images, including for dental applications. Currently, the problem with dental applications is the wide variety of incompatible file formats, for which no single neutral storage exists, and for which physicians must use many different viewers: DICOM should be the solution to break these proprietary, closed, non-competitive ecosystems.
Any 2D x-rays and 3D x-rays imaging device can already be configured to generate DICOM files. 2D visible-light images and 2D films can be encapsulated as DICOM images (a process known as “DICOM-ization”, for which free and open-source tools do exist, such as Orthanc through its REST API, or GDCM and DCMTK through their command-line tools). DICOM primitives to encode digital dental casts do exist, but the FOSS community should unite to work on a reference implementation to convert from STL to DICOM (in particular, the Orthanc community would be happy to contribute to this effort). Finally, as far as 3D films are concerned, DICOM supports stereoscopic movies.
If you have any question regarding Orthanc, feel free to get in touch with me.