It is accepted in the Linked Data for Architecture and Construction (LDAC) community that generating knowledge graphs (KGs) from the BIM model of a building enables higher level use cases such as integration with geographic information systems, operational system integration, semantic digital twins (DTs), or automatic compliance checking. However, existing approaches generate a large, monolithic knowledge graph that is difficult to integrate with other knowledge such as Thing Descriptions (TDs) of Internet of Things (IoT) devices, or information about office occupants and room occupancy schedules. In this work, we describe a set of three modular knowledge graphs that enable knowledge integration for the semantic DT of our building at Mines Saint-Étienne, leveraging the principles of Linked Building Data: (1) KG LBD is automatically generated from the Revit model of our building, (2) KG FOAF is semi-automatically generated from the employee directory of Mines Saint-Étienne, and (3) KG TD is automatically generated from the ETS5 project file describing the KNX network in our building using the W3C TD ontology, and points to real-time and historical data. Our approach offers an alternative with respect to the state of the art such that: (1) relevant bits of the building's KG can be accessed using a simple REST-like interface, where each small KG contains links to other entities that themselves are identified by an IRI and have a small KG accessible; (2) Knowledge potentially served by different servers can be integrated in the same solution; (3) simple access control can be implemented for some parts of the global KG.