{"id":463,"date":"2012-11-29T14:27:00","date_gmt":"2012-11-29T14:27:00","guid":{"rendered":"\/blogs\/bobb\/post\/Loading-spatial-data-and-cleansing-converting-with-SQL-Server-2012.aspx"},"modified":"2014-12-30T13:03:44","modified_gmt":"2014-12-30T21:03:44","slug":"loading-spatial-data-and-cleansing-converting-with-sql-server-2012","status":"publish","type":"post","link":"https:\/\/www.sqlskills.com\/blogs\/bobb\/loading-spatial-data-and-cleansing-converting-with-sql-server-2012\/","title":{"rendered":"Loading spatial data and cleansing-converting with SQL Server 2012"},"content":{"rendered":"<p>\nSomeone asked, on Twitter, if it were possible to get a set of spatial data that included cities, counties (administrative regions) and countries in database table format for SQL Server spatial. I&#39;d come across just these datasets (and more) as shapefiles at the CDC website and pointed them out. Then came the question about how to load these shapefiles into SQL Server.\n<\/p>\n<p>\nI&#39;ve always preferred FME by Safe Software to load spatial data into SQL Server. They have <a href=\"http:\/\/www.safe.com\/solutions\/for-databases\/microsoft-sql-server\/\">a variety of products <\/a>&nbsp;for SQL Server, including a low-priced option that more than suffices for loading shapefiles. But what if you can&#39;t afford even Safe&#39;s low-priced option (you&#39;re doing it for proof-of-concept, for example). I&#39;d then recommend Shape2SQL, a free tool that&#39;s been around since SQL Server Spatial&#39;s inception back in SQL Server 2008.\n<\/p>\n<p>\nThe first time I&#39;d loaded these shapefiles let&#39;s use the cntry00 (countries) file as an example, I&#39;d run into some problems&nbsp;loading them as geography data type and had to bypass some records in the shapefile, leaving me with a map that was missing some countries. I&#39;d always assumed that this was a ring-order problem that produced geographies large than a hemisphere. SQL Server Spatial in 2012 fixed the &quot;larger than a hemisphere&quot; limitation, so I thought I&#39;d try loading as a geography again. Unfortunately I got the same errors. So I decided to reload the data as geometry, cleanse the data and convert to geography. What follows is a series of steps to do just that. It relies on some SQL Server 2012-specific functionality.\n<\/p>\n<p>\nAfter trying to summarize this on Twitter as 140-character messages (I always refer to Twitter as &quot;my 140-character lobotomy&quot;) someone suggested this would be a good blog entry. So here it is.\n<\/p>\n<p>\nSo, acquire the shapefile data. What you&#39;re looking for is a .shp file (input into Shape2SQL) and possibly the .prj file to determine the SRID to use. In this case, I&#39;ll go with SRID 4326 eventually for geography. You can use an SRID with geometry too, but it doesn&#39;t effect the spatial library&#39;s computation like it does with geography.\n<\/p>\n<p>\n1. Unzip the zipfile<br \/>\n2. Pointed Shape2SQL at the .shp file choosing the following options:<br \/>\n&nbsp;&nbsp; a. Planar Geometry<br \/>\n&nbsp;&nbsp; b. Set SRID 4326<br \/>\n&nbsp;&nbsp; c. Create spatial index &#8211; no (checkbox cleared)<br \/>\n&nbsp;&nbsp; d. spatial column name &#8211; geom (default)\n<\/p>\n<p>\nNow all the data will load. So let&#39;s run some queries to clean things up and convert to geography.\n<\/p>\n<p>\n&#8212; First, look for Invalid geometries and get the reasons why they are invalid<br \/>\n&#8212; IsValidDetailed is SQL Server 2012-specific<br \/>\n&#8212; BTW, it wasn&#39;t a ring order problem after all.\n<\/p>\n<p>\n&#8212; Error: 24411<br \/>\nselect geom.IsValidDetailed(), * from cntry00<br \/>\nwhere geom.STIsValid() = 0;\n<\/p>\n<p>\n&#8212; Make them valid, noting the row ids for your reference<br \/>\n&#8212; rows 21, 42, 252<br \/>\nupdate cntry00<br \/>\nset geom = geom.MakeValid()<br \/>\noutput inserted.id<br \/>\nwhere geom.STIsValid() = 0;\n<\/p>\n<p>\n&#8212; add geography column<br \/>\nalter table cntry00 add geog geography;\n<\/p>\n<p>\n&#8212; vacuous conversion to geography<br \/>\nupdate cntry00<br \/>\nset geog = geography::STGeomFromWKB(geom.STAsBinary(), 4326);\n<\/p>\n<p>\n&#8212; see if any invalid now&#8230; and the reason<br \/>\n&#8212; 24413, 24414 near the poles (3 rows)<br \/>\nselect geog.IsValidDetailed(), * from cntry00<br \/>\nwhere geog.STIsValid() = 0;\n<\/p>\n<p>\n&#8212; validate again<br \/>\n&#8212; rows 1,181,215<br \/>\nupdate cntry00<br \/>\nset geog = geog.MakeValid()<br \/>\noutput inserted.id<br \/>\nwhere geog.STIsValid() = 0;\n<\/p>\n<p>\n&#8212; check for more than a hemisphere, none found<br \/>\nselect * from cntry00<br \/>\nwhere geog.EnvelopeAngle() &gt; 90;\n<\/p>\n<p>\n&#8212; if there are geographies greater than a hemisphere <br \/>\n&#8212; because of ring order problems fix &#39;em<br \/>\n&#8212; SQL Server 2012-specific<br \/>\nupdate cntry00<br \/>\n&nbsp;set geog = geog.ReorientObject()<br \/>\n&nbsp;output inserted.id<br \/>\n&nbsp;where geog.EnvelopeAngle() &gt; 90;\n<\/p>\n<p>\nAnd there you have it. You might think that you can use the spatial results tab in SSMS to compare your data before and after validation to see if validation has left artifacts or changed the shape. Don&#39;t bother, though; the spatial results tab called MakeValid before displaying the data.\n<\/p>\n<p>\nHope this helps,<br \/>\nBob\n<\/p>\n<p>\n@bobbeauch<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Someone asked, on Twitter, if it were possible to get a set of spatial data that included cities, counties (administrative regions) and countries in database table format for SQL Server spatial. I&#39;d come across just these datasets (and more) as shapefiles at the CDC website and pointed them out. Then came the question about how [&hellip;]<\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[31,36],"tags":[],"class_list":["post-463","post","type-post","status-publish","format-standard","hentry","category-sql-server-2012","category-sql-server-spatial"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.9.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Loading spatial data and cleansing-converting with SQL Server 2012 - Bob Beauchemin<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.sqlskills.com\/blogs\/bobb\/loading-spatial-data-and-cleansing-converting-with-sql-server-2012\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Loading spatial data and cleansing-converting with SQL Server 2012 - Bob Beauchemin\" \/>\n<meta property=\"og:description\" content=\"Someone asked, on Twitter, if it were possible to get a set of spatial data that included cities, counties (administrative regions) and countries in database table format for SQL Server spatial. I&#039;d come across just these datasets (and more) as shapefiles at the CDC website and pointed them out. Then came the question about how [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.sqlskills.com\/blogs\/bobb\/loading-spatial-data-and-cleansing-converting-with-sql-server-2012\/\" \/>\n<meta property=\"og:site_name\" content=\"Bob Beauchemin\" \/>\n<meta property=\"article:published_time\" content=\"2012-11-29T14:27:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2014-12-30T21:03:44+00:00\" \/>\n<meta name=\"author\" content=\"Bob Beauchemin\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Bob Beauchemin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.sqlskills.com\/blogs\/bobb\/loading-spatial-data-and-cleansing-converting-with-sql-server-2012\/\",\"url\":\"https:\/\/www.sqlskills.com\/blogs\/bobb\/loading-spatial-data-and-cleansing-converting-with-sql-server-2012\/\",\"name\":\"Loading spatial data and cleansing-converting with SQL Server 2012 - Bob Beauchemin\",\"isPartOf\":{\"@id\":\"https:\/\/www.sqlskills.com\/blogs\/bobb\/#website\"},\"datePublished\":\"2012-11-29T14:27:00+00:00\",\"dateModified\":\"2014-12-30T21:03:44+00:00\",\"author\":{\"@id\":\"https:\/\/www.sqlskills.com\/blogs\/bobb\/#\/schema\/person\/62bfa986c5b5d28fcffd8b4fc409c73e\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.sqlskills.com\/blogs\/bobb\/loading-spatial-data-and-cleansing-converting-with-sql-server-2012\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.sqlskills.com\/blogs\/bobb\/loading-spatial-data-and-cleansing-converting-with-sql-server-2012\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.sqlskills.com\/blogs\/bobb\/loading-spatial-data-and-cleansing-converting-with-sql-server-2012\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.sqlskills.com\/blogs\/bobb\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"SQL Server 2012\",\"item\":\"https:\/\/www.sqlskills.com\/blogs\/bobb\/category\/sql-server-2012\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Loading spatial data and cleansing-converting with SQL Server 2012\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.sqlskills.com\/blogs\/bobb\/#website\",\"url\":\"https:\/\/www.sqlskills.com\/blogs\/bobb\/\",\"name\":\"Bob Beauchemin\",\"description\":\"SQL Server Blog\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.sqlskills.com\/blogs\/bobb\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.sqlskills.com\/blogs\/bobb\/#\/schema\/person\/62bfa986c5b5d28fcffd8b4fc409c73e\",\"name\":\"Bob Beauchemin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.sqlskills.com\/blogs\/bobb\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/6f80e6cc667410857fa6a21931dc528b8092f4d112bf7a8ff7c267674d44ee37?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/6f80e6cc667410857fa6a21931dc528b8092f4d112bf7a8ff7c267674d44ee37?s=96&d=mm&r=g\",\"caption\":\"Bob Beauchemin\"},\"sameAs\":[\"http:\/www.sqlskills.com\/blogs\/bobb\/\"],\"url\":\"https:\/\/www.sqlskills.com\/blogs\/bobb\/author\/bobb\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Loading spatial data and cleansing-converting with SQL Server 2012 - Bob Beauchemin","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.sqlskills.com\/blogs\/bobb\/loading-spatial-data-and-cleansing-converting-with-sql-server-2012\/","og_locale":"en_US","og_type":"article","og_title":"Loading spatial data and cleansing-converting with SQL Server 2012 - Bob Beauchemin","og_description":"Someone asked, on Twitter, if it were possible to get a set of spatial data that included cities, counties (administrative regions) and countries in database table format for SQL Server spatial. I&#39;d come across just these datasets (and more) as shapefiles at the CDC website and pointed them out. Then came the question about how [&hellip;]","og_url":"https:\/\/www.sqlskills.com\/blogs\/bobb\/loading-spatial-data-and-cleansing-converting-with-sql-server-2012\/","og_site_name":"Bob Beauchemin","article_published_time":"2012-11-29T14:27:00+00:00","article_modified_time":"2014-12-30T21:03:44+00:00","author":"Bob Beauchemin","twitter_misc":{"Written by":"Bob Beauchemin","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.sqlskills.com\/blogs\/bobb\/loading-spatial-data-and-cleansing-converting-with-sql-server-2012\/","url":"https:\/\/www.sqlskills.com\/blogs\/bobb\/loading-spatial-data-and-cleansing-converting-with-sql-server-2012\/","name":"Loading spatial data and cleansing-converting with SQL Server 2012 - Bob Beauchemin","isPartOf":{"@id":"https:\/\/www.sqlskills.com\/blogs\/bobb\/#website"},"datePublished":"2012-11-29T14:27:00+00:00","dateModified":"2014-12-30T21:03:44+00:00","author":{"@id":"https:\/\/www.sqlskills.com\/blogs\/bobb\/#\/schema\/person\/62bfa986c5b5d28fcffd8b4fc409c73e"},"breadcrumb":{"@id":"https:\/\/www.sqlskills.com\/blogs\/bobb\/loading-spatial-data-and-cleansing-converting-with-sql-server-2012\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.sqlskills.com\/blogs\/bobb\/loading-spatial-data-and-cleansing-converting-with-sql-server-2012\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.sqlskills.com\/blogs\/bobb\/loading-spatial-data-and-cleansing-converting-with-sql-server-2012\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.sqlskills.com\/blogs\/bobb\/"},{"@type":"ListItem","position":2,"name":"SQL Server 2012","item":"https:\/\/www.sqlskills.com\/blogs\/bobb\/category\/sql-server-2012\/"},{"@type":"ListItem","position":3,"name":"Loading spatial data and cleansing-converting with SQL Server 2012"}]},{"@type":"WebSite","@id":"https:\/\/www.sqlskills.com\/blogs\/bobb\/#website","url":"https:\/\/www.sqlskills.com\/blogs\/bobb\/","name":"Bob Beauchemin","description":"SQL Server Blog","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.sqlskills.com\/blogs\/bobb\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.sqlskills.com\/blogs\/bobb\/#\/schema\/person\/62bfa986c5b5d28fcffd8b4fc409c73e","name":"Bob Beauchemin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.sqlskills.com\/blogs\/bobb\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/6f80e6cc667410857fa6a21931dc528b8092f4d112bf7a8ff7c267674d44ee37?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/6f80e6cc667410857fa6a21931dc528b8092f4d112bf7a8ff7c267674d44ee37?s=96&d=mm&r=g","caption":"Bob Beauchemin"},"sameAs":["http:\/www.sqlskills.com\/blogs\/bobb\/"],"url":"https:\/\/www.sqlskills.com\/blogs\/bobb\/author\/bobb\/"}]}},"_links":{"self":[{"href":"https:\/\/www.sqlskills.com\/blogs\/bobb\/wp-json\/wp\/v2\/posts\/463","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sqlskills.com\/blogs\/bobb\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sqlskills.com\/blogs\/bobb\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sqlskills.com\/blogs\/bobb\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sqlskills.com\/blogs\/bobb\/wp-json\/wp\/v2\/comments?post=463"}],"version-history":[{"count":0,"href":"https:\/\/www.sqlskills.com\/blogs\/bobb\/wp-json\/wp\/v2\/posts\/463\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.sqlskills.com\/blogs\/bobb\/wp-json\/wp\/v2\/media?parent=463"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sqlskills.com\/blogs\/bobb\/wp-json\/wp\/v2\/categories?post=463"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sqlskills.com\/blogs\/bobb\/wp-json\/wp\/v2\/tags?post=463"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}