{"id":14429,"date":"2022-03-02T16:13:40","date_gmt":"2022-03-02T15:13:40","guid":{"rendered":"https:\/\/www.qintesi.com\/?post_type=portfolio&p=14429"},"modified":"2022-05-05T10:15:09","modified_gmt":"2022-05-05T08:15:09","slug":"enterprise-data-platform-with-google-cloud","status":"publish","type":"portfolio","link":"https:\/\/www.qintesi.com\/en\/portfolio\/enterprise-data-platform-with-google-cloud\/","title":{"rendered":"Enterprise data platform with Google Cloud"},"content":{"rendered":"
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Challenge<\/h4>

The client, a leading company in the<\/span>\u00a0marketing and digital advertising sector<\/b>, has started an\u00a0<\/span>IT transformation<\/b>\u00a0program aimed at creating a new data platform with\u00a0<\/span>Google Cloud Platform<\/b>\u00a0to which to migrate the existing data warehouse and reporting applications, to achieve the following objectives:<\/span><\/p>

  • Adopting a new flexible, modular and scalable data platform, simplifying the current architecture by introducing a cloud-based technology stack<\/span><\/li>
  • Improving data quality<\/span><\/li>
  • Creating a unified data view that includes all the entities with which it is possible to describe the company business and monitor its performance<\/span><\/li>
  • Anonymizing sensitive data according to the GDPR regulations<\/span><\/li><\/ul>
    \u00a0<\/div>

    Solution<\/h4>

    For the implementation of the data platform,<\/span>\u00a0Google Cloud Storage<\/b>\u00a0was used as a data lake and\u00a0<\/span>Google BigQuery\u00a0<\/b>as a data warehouse. The data was organized into zones:<\/span><\/p>

    • In the<\/span>\u00a0Raw Zone t<\/b>here are the raw data divided, based on their use, in landing (data not yet processed), archived (data already successfully processed and archived), and invalid (data processed without success), in addition to data relating to log in the Logging zone.<\/span><\/li>
    • In the<\/span>\u00a0Processed Zone<\/b>\u00a0on BigQuery you will find structured data that has successfully passed a first level of cleaning and normalization.<\/span><\/li>
    • The\u00a0<\/span>Refined Zone<\/b>\u00a0contains the transformed data in the unified view, available for analysis and reporting needs<\/span><\/li><\/ul>

      The data loading system has been implemented in a flexible, modular and scalable way, allowing the addition of new flows and\/or source systems with minimal impact.<\/span><\/p>

      Flexibility\u00a0<\/b>also characterizes the construction phase of the views for\u00a0<\/span>reporting<\/b>, which provides for the possibility of adding new KPIs in a parametric way: in this way the structures are automatically updated during the release phase, without the need for manual changes to the code.<\/span><\/p>

      In the Refined Zone there are also predictive models, the results of which are in turn available for further analysis (e.g. customer churn prediction).<\/span><\/p>

      The scheduling of processes is managed with\u00a0<\/span>Cloud Composer<\/b>, while for their monitoring, including the verification of data quality, alerting thresholds have been defined with automatic sending of emails via\u00a0<\/span>Cloud Logging<\/b>; for more in-depth analysis, a dashboard created with\u00a0<\/span>Google Data Studio<\/b>\u00a0is available.<\/span><\/p>

      To support Governance, to ensure the traceability of information from the source systems to the views for reporting, the data lineage function has been implemented, accessible from the\u00a0<\/span>Google Data Catalog<\/b>\u00a0service, from which it is also possible to access the description and other information of all the entities (tables, fields, files\u2026) present in the data platform.<\/span><\/p>

      During implementation, the utmost attention was always paid to cost control, through continuous monitoring of the impact of the released code and verification of the application of best practices.<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

      \n\t\t\t\t\t\t
      \n\t\t\t\t\t
      \n\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t

      Results<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t
        \n\t\t\t\t\t\t\t
      • \n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t<\/i>\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\tComplete review of all corporate performance monitoring indices through the definition of KPIs<\/span>\n\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t
      • \n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t<\/i>\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\tReengineering of the on-premise data warehouse on Google BigQuery<\/span>\n\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t
      • \n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t<\/i>\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\tSpeed up of loading and transformation processes<\/span>\n\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t
      • \n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t<\/i>\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\tReduction of the time required to make data available for reporting<\/span>\n\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t
      • \n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t<\/i>\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\tIntegration into the data warehouse of the new Planning, Finance, HR, CRM, Commissioning, Campaign Management and Website Analysis systems<\/span>\n\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t\t\t
      • \n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t<\/i>\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\t\tAdoption of the CI\/CD methodology for automation and continuous monitoring of the code life cycle together with a \"Test Driven Development\" approach<\/span>\n\t\t\t\t\t\t\t\t\t<\/li>\n\t\t\t\t\t\t<\/ul>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
        \n\t\t\t\t\t\t
        \n\t\t\t\t\t
        \n\t\t\t
        \n\t\t\t\t\t\t
        \n\t\t\t\t
        \n\t\t\t\t\t\t\t\t\t

        Le informazioni contenute in questo documento sono di propriet\u00e0.
        Copyright \u00a9 2014 Qintesi S.p.A. Tutti i diritti riservati.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>","protected":false},"excerpt":{"rendered":"

        Challenge The client, a leading company in the\u00a0marketing and digital advertising sector, has started an\u00a0IT transformation\u00a0program aimed at creating a new data platform with\u00a0Google Cloud Platform\u00a0to which to migrate the existing data warehouse and reporting applications, to achieve the following objectives: Adopting a new flexible, modular and scalable data platform, simplifying the current architecture by […]<\/p>","protected":false},"author":3,"featured_media":18452,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","format":"standard","meta":{"_vp_format_video_url":"","_vp_image_focal_point":[],"footnotes":""},"portfolio_category":[],"portfolio_tag":[762,763,764],"servizi-offerti":[808],"moduli_implementati":[],"settore":[831],"class_list":["post-14429","portfolio","type-portfolio","status-publish","format-standard","has-post-thumbnail","hentry","portfolio_tag-cloud-en-2","portfolio_tag-google-en-2","portfolio_tag-marketing-en-2","servizi-offerti-digital-transformation","settore-professional-services-en"],"yoast_head":"\nEnterprise data platform with Google Cloud - Qintesi<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.qintesi.com\/portfolio\/enterprise-data-platform-con-google-cloud\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Enterprise data platform with Google Cloud - Qintesi\" \/>\n<meta property=\"og:description\" content=\"Challenge The client, a leading company in the\u00a0marketing and digital advertising sector, has started an\u00a0IT transformation\u00a0program aimed at creating a new data platform with\u00a0Google Cloud Platform\u00a0to which to migrate the existing data warehouse and reporting applications, to achieve the following objectives: Adopting a new flexible, modular and scalable data platform, simplifying the current architecture by […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.qintesi.com\/portfolio\/enterprise-data-platform-con-google-cloud\/\" \/>\n<meta property=\"og:site_name\" content=\"Qintesi\" \/>\n<meta property=\"article:modified_time\" content=\"2022-05-05T08:15:09+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.qintesi.com\/wp-content\/uploads\/2022\/03\/Enterprise-data-platform-con-Google-Cloud_Miniatura.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"876\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.qintesi.com\\\/portfolio\\\/enterprise-data-platform-con-google-cloud\\\/\",\"url\":\"https:\\\/\\\/www.qintesi.com\\\/portfolio\\\/enterprise-data-platform-con-google-cloud\\\/\",\"name\":\"Enterprise data platform with Google Cloud - Qintesi\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.qintesi.com\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.qintesi.com\\\/portfolio\\\/enterprise-data-platform-con-google-cloud\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.qintesi.com\\\/portfolio\\\/enterprise-data-platform-con-google-cloud\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.qintesi.com\\\/wp-content\\\/uploads\\\/2022\\\/03\\\/Enterprise-data-platform-con-Google-Cloud_Miniatura.png\",\"datePublished\":\"2022-03-02T15:13:40+00:00\",\"dateModified\":\"2022-05-05T08:15:09+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.qintesi.com\\\/portfolio\\\/enterprise-data-platform-con-google-cloud\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.qintesi.com\\\/portfolio\\\/enterprise-data-platform-con-google-cloud\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.qintesi.com\\\/portfolio\\\/enterprise-data-platform-con-google-cloud\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.qintesi.com\\\/wp-content\\\/uploads\\\/2022\\\/03\\\/Enterprise-data-platform-con-Google-Cloud_Miniatura.png\",\"contentUrl\":\"https:\\\/\\\/www.qintesi.com\\\/wp-content\\\/uploads\\\/2022\\\/03\\\/Enterprise-data-platform-con-Google-Cloud_Miniatura.png\",\"width\":1920,\"height\":876},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.qintesi.com\\\/portfolio\\\/enterprise-data-platform-con-google-cloud\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.qintesi.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Portfolio Items\",\"item\":\"https:\\\/\\\/www.qintesi.com\\\/portfolio\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Enterprise data platform with Google Cloud\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.qintesi.com\\\/#website\",\"url\":\"https:\\\/\\\/www.qintesi.com\\\/\",\"name\":\"Qintesi\",\"description\":\"People shaping solutions.\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.qintesi.com\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.qintesi.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.qintesi.com\\\/#organization\",\"name\":\"Qintesi\",\"url\":\"https:\\\/\\\/www.qintesi.com\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.qintesi.com\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.qintesi.com\\\/wp-content\\\/uploads\\\/2022\\\/03\\\/qintesi_logo.png\",\"contentUrl\":\"https:\\\/\\\/www.qintesi.com\\\/wp-content\\\/uploads\\\/2022\\\/03\\\/qintesi_logo.png\",\"width\":908,\"height\":681,\"caption\":\"Qintesi\"},\"image\":{\"@id\":\"https:\\\/\\\/www.qintesi.com\\\/#\\\/schema\\\/logo\\\/image\\\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Enterprise data platform with Google Cloud - Qintesi","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.qintesi.com\/portfolio\/enterprise-data-platform-con-google-cloud\/","og_locale":"en_US","og_type":"article","og_title":"Enterprise data platform with Google Cloud - Qintesi","og_description":"Challenge The client, a leading company in the\u00a0marketing and digital advertising sector, has started an\u00a0IT transformation\u00a0program aimed at creating a new data platform with\u00a0Google Cloud Platform\u00a0to which to migrate the existing data warehouse and reporting applications, to achieve the following objectives: Adopting a new flexible, modular and scalable data platform, simplifying the current architecture by […]","og_url":"https:\/\/www.qintesi.com\/portfolio\/enterprise-data-platform-con-google-cloud\/","og_site_name":"Qintesi","article_modified_time":"2022-05-05T08:15:09+00:00","og_image":[{"width":1920,"height":876,"url":"https:\/\/www.qintesi.com\/wp-content\/uploads\/2022\/03\/Enterprise-data-platform-con-Google-Cloud_Miniatura.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.qintesi.com\/portfolio\/enterprise-data-platform-con-google-cloud\/","url":"https:\/\/www.qintesi.com\/portfolio\/enterprise-data-platform-con-google-cloud\/","name":"Enterprise data platform with Google Cloud - Qintesi","isPartOf":{"@id":"https:\/\/www.qintesi.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.qintesi.com\/portfolio\/enterprise-data-platform-con-google-cloud\/#primaryimage"},"image":{"@id":"https:\/\/www.qintesi.com\/portfolio\/enterprise-data-platform-con-google-cloud\/#primaryimage"},"thumbnailUrl":"https:\/\/www.qintesi.com\/wp-content\/uploads\/2022\/03\/Enterprise-data-platform-con-Google-Cloud_Miniatura.png","datePublished":"2022-03-02T15:13:40+00:00","dateModified":"2022-05-05T08:15:09+00:00","breadcrumb":{"@id":"https:\/\/www.qintesi.com\/portfolio\/enterprise-data-platform-con-google-cloud\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.qintesi.com\/portfolio\/enterprise-data-platform-con-google-cloud\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.qintesi.com\/portfolio\/enterprise-data-platform-con-google-cloud\/#primaryimage","url":"https:\/\/www.qintesi.com\/wp-content\/uploads\/2022\/03\/Enterprise-data-platform-con-Google-Cloud_Miniatura.png","contentUrl":"https:\/\/www.qintesi.com\/wp-content\/uploads\/2022\/03\/Enterprise-data-platform-con-Google-Cloud_Miniatura.png","width":1920,"height":876},{"@type":"BreadcrumbList","@id":"https:\/\/www.qintesi.com\/portfolio\/enterprise-data-platform-con-google-cloud\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.qintesi.com\/"},{"@type":"ListItem","position":2,"name":"Portfolio Items","item":"https:\/\/www.qintesi.com\/portfolio\/"},{"@type":"ListItem","position":3,"name":"Enterprise data platform with Google Cloud"}]},{"@type":"WebSite","@id":"https:\/\/www.qintesi.com\/#website","url":"https:\/\/www.qintesi.com\/","name":"Qintesi","description":"People shaping solutions.","publisher":{"@id":"https:\/\/www.qintesi.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.qintesi.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.qintesi.com\/#organization","name":"Qintesi","url":"https:\/\/www.qintesi.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.qintesi.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.qintesi.com\/wp-content\/uploads\/2022\/03\/qintesi_logo.png","contentUrl":"https:\/\/www.qintesi.com\/wp-content\/uploads\/2022\/03\/qintesi_logo.png","width":908,"height":681,"caption":"Qintesi"},"image":{"@id":"https:\/\/www.qintesi.com\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/www.qintesi.com\/en\/wp-json\/wp\/v2\/portfolio\/14429","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.qintesi.com\/en\/wp-json\/wp\/v2\/portfolio"}],"about":[{"href":"https:\/\/www.qintesi.com\/en\/wp-json\/wp\/v2\/types\/portfolio"}],"author":[{"embeddable":true,"href":"https:\/\/www.qintesi.com\/en\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.qintesi.com\/en\/wp-json\/wp\/v2\/comments?post=14429"}],"version-history":[{"count":0,"href":"https:\/\/www.qintesi.com\/en\/wp-json\/wp\/v2\/portfolio\/14429\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.qintesi.com\/en\/wp-json\/wp\/v2\/media\/18452"}],"wp:attachment":[{"href":"https:\/\/www.qintesi.com\/en\/wp-json\/wp\/v2\/media?parent=14429"}],"wp:term":[{"taxonomy":"portfolio_category","embeddable":true,"href":"https:\/\/www.qintesi.com\/en\/wp-json\/wp\/v2\/portfolio_category?post=14429"},{"taxonomy":"portfolio_tag","embeddable":true,"href":"https:\/\/www.qintesi.com\/en\/wp-json\/wp\/v2\/portfolio_tag?post=14429"},{"taxonomy":"servizi-offerti","embeddable":true,"href":"https:\/\/www.qintesi.com\/en\/wp-json\/wp\/v2\/servizi-offerti?post=14429"},{"taxonomy":"moduli_implementati","embeddable":true,"href":"https:\/\/www.qintesi.com\/en\/wp-json\/wp\/v2\/moduli_implementati?post=14429"},{"taxonomy":"settore","embeddable":true,"href":"https:\/\/www.qintesi.com\/en\/wp-json\/wp\/v2\/settore?post=14429"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}