Data Architecture, Data Modelling and Data Processing (Notes)

Data Architecture (DA)

Most organisations / companies have enterprise architecture which describes how a computer network (with storage, servers, switches, routers etc.) is laid out. Data architecture is very similar in that it shows the models, rules, standards, and policies that are used in the collection, storage, and use of data in an organisation / company. The data architecture diagram highlights data / data flow more than a regular systems architecture diagram.

Components that can be included in a data architecture diagram include storage (database, warehouse, lake), pipelines, APIs and analytics software.

A data architect takes business requirements, transforms them into technical specifications and visual data management frameworks. This includes defining the data framework, standards, flows (who generates data, how the data transforms, where the data flows to) and collaborating with various others (stakeholders, partners, vendors).

A data architect focuses on the tools and platforms that will be used in the data architecture, and the security of the data.

Data Modelling (DM)

Unlike a DA, a Data Modeller focuses on the modelling (representation), accuracy and reliability of the data. The viewpoint on the term “data model” varies between Data Scientists, Data Engineers, Statisticians and Data Analysts.

Batch Processing vs Real-Time Processing

Batch processing involves data being processed in large batches (asynchronously). Real-time processing is the opposite of batch processing, as data is processed as it is received. The current trend is a move from batch processing to real-time processing, with a trend of moving from on-premises computing to cloud computing.