Posts

Recursively Delete files of the Same Name in S3

  From the CLI you can delete all files recursively of the same name (/mydata.csv) using This is for cases where a file of the same name appears in subfolders of a bucket and you want to delete all occurences of that file. If you have bucket versioning turned on then you will not actually lose the file forever. aws s3 rm s3://mybucket/toplevelfolder/ --recursive --exclude "*" --include "*/mydata.csv" If you are paranoid about screwing things up there’s an optional – dryrun parameter that generates a report of what will be deleted, without actually deleting.   aws s3 rm s3:// mybucket/toplevelfolder / --dryrun --recursive --exclude "*" --include "*/metadata.csv" (dryrun) delete: s3:// mybucket/toplevelfolder /part_2/mydata.csv (dryrun) delete: s3:// mybucket/toplevelfolder /part_3/mydata.csv  
Post a Comment
Read more

RedShift - Renaming - ALTER TABLE

Rename a table The following command renames the USERS table to USERS_BKUP: alter table users rename to users_bkup; You can also use this type of command to rename a view. Change the owner of a table or view The following command changes the VENUE table owner to the user DWUSER: alter table venue owner to dwuser; Rename a column The following command renames the VENUESEATS column in the VENUE table to VENUESIZE: alter table venue rename column venueseats to venuesize;
Post a Comment
Read more

Redshift - Show Tables in schema

select t.table_name from information_schema.tables t where t.table_schema = 'myschema' -- put your schema name here and t.table_type = 'BASE TABLE' order by t.table_name;
Post a Comment
Read more

Python - datetime Formatting

Formatting datetime in Python can be a 'pain'. Formatting works in two directions - From a string to datetime. - From a datetime to string. The easiest way to remember how things work using the strptime and strftime is that: strptime (string)(parse)(time) strftime (string)(format)(time) Format a datetime to a string from datetime import datetime print(datetime.today( ).strftime('%Y-%m-%d %H:%M:%S')) 2020-07-22 10:07:16 Parse a String to datetime from datetime import datetime date_time = ' 2020-07-22 10:07:16 ' dt_object =  datetime .strptime(date_time, ' %Y-%m-%d %H:%M:%S' )
Post a Comment
Read more

Rename Key in S3 (Python)

S3 has no rename functionality. You need to copy to a new 'file' and delete the original. In python using boto3 s3 = boto3 . resource ( 's3' ) s3 . Object ( 'my_bucket' , 'my_file_new' ). copy_from ( CopySource = 'my_bucket/my_file_old' ) s3 . Object ( 'my_bucket' , 'my_file_old' ). delete ()
Post a Comment
Read more

ORACLE RDS granting select privileges to users.

On the Oracle RDS service in AWS you have no real SYS/SYSDBA account or physical machine. So how do you grant select on SYS objects? AWS have the RDSADMIN account with some useful packages and I have used one of the procedures in RDSADMIN_UTIL to do the job. -- Granting Permissions to SYS Objects in AWS RDS -- In RDS you do not have access to the server or to a proper SYSDBA account.  -- begin     rdsadmin . rdsadmin_util . grant_sys_object ( p_obj_name => ' V_$SESSION ' ,         p_grantee => 'DASHVIEW ' ,         p_privilege => ' SELECT ' ) ; end ; /
Post a Comment
Read more

S3 Doesn't Have Folders!

The S3 folder delusion When you AWS console you can create folders to group objects but this is just a delusion deliberately created by AWS to simplify the usage. In reality, S3 has a flat structure and all the objects are on the same level. Here’s the excerpt from AWS documentation that states this fact: In Amazon S3, buckets and objects are the primary resources, where objects are stored in buckets. Amazon S3 has a flat structure with no hierarchy like you would see in a typical file system. However, for the sake of organizational simplicity, the Amazon S3 console supports the folder concept as a means of grouping objects. Amazon S3 does this by using key name prefixes for objects. So essentially AWS is just smart enough to recognize the standard folder notation we’ve been using for ages to make this things easier for us. All about S3 objects.
Post a Comment
Read more